Enterprise AI Gateway: Four Architectural Principles

W2 covered trust — "the protocol lets agents correct each other." This week is the next layer of that protocol — how to collapse chaos into an abstraction layer.

Another moment from that 2-hour hospital relay:

13:30-13:50 — voice polish + cross-week scaffolding

Erwin did two things in those 20 minutes: (1) caught 11 "doesn't sound human" sentences in the W1 cornerstone and rewrote them in conversational tone; (2) rewrote the opening of W1 Section 4 into "three cross-role collaboration pairs," each pair seeding a foreshadow that points to W2/W3/W4 later.

The key to doing both in 20 minutes wasn't "Erwin is smart" — it was 1M context window + Turn-boundary batching letting him hold all of W1, the W2/W3/W4 drafts, plus the voice feedback from earlier in the thread, all at once. All the chaos got collapsed into one context layer.

That's exactly what an enterprise AI gateway has to solve — not one agent, but how N agents / N LLMs / N MCPs of chaos collapse into a single layer. The four principles below are the concrete shape of that collapse.

What makes an AI gateway enterprise-ready? Not features. Principles.

The moment your company runs a second AI agent, a second LLM vendor, or a first third-party MCP server, the "enterprise AI gateway" problem appears: who consolidates this AI traffic? How do you guarantee data doesn't quietly exfiltrate? Do you rewrite every call site the next time you swap vendors?

I've spent the last several months evaluating Cloudflare AI Gateway, Portkey, LiteLLM, Helicone, OpenRouter, and each vendor's own enterprise tier against real, data-sensitive enterprise environments. What I keep noticing is the same gap — most of these solutions miss the same thing: the feature lists are dense, but the underlying architectural choices don't line up.

The gap isn't in feature lists. It's in four foundational architectural choices. If those four aren't done right at the same time, the "enterprise AI gateway" amplifies attack surface instead of compressing it. Every AI coding agent promises speed — but when your CIO asks "how do we control this?", a feature list doesn't answer. Principles do.

The four below are my current read — not industry consensus, but where I've landed after evaluating the market:

Principle 1: Control by Configuration

Policy as code, not as convention.

Don't tell engineers to "be careful" with AI agents. Codify it.

# One file governs everything
[discord]
allowed_channels = ["frontend-ai"]    # channel boundary
trusted_bot_ids = ["erwin", "hange"]  # whitelist

[agent]
working_dir = "/app/frontend"         # file access scope
env = { NODE_ENV = "development" }    # env whitelist

[identity]
credential_ttl = "15m"                # short-lived credential
capability_scope = ["erp.read", "mes.query"]  # least privilege

Every control is in config.toml. No database. No admin panel. No "ask DevOps to change it."

This matters disproportionately in the agentic era. The industry median ratio of non-human identities (NHI) to human identities is 45:1 (Rubrik Zero Labs 2026), and 144:1 in DevOps environments. 53% of organizations have agents holding privileges beyond their original design (CSA / TierZero 2026). The traditional pattern — each agent carrying its own long-lived API key to ERP / MES / CRM — means a single agent credential leak equals simultaneous compromise across multiple systems.

The right design: agents obtain short-lived, capability-scoped credentials through the gateway. Credentials don't leave the gateway; agents only get a session token. This maps onto Norm Hardy's 1988 confused deputy model — when a deputy (an agent) holds multiple capabilities, an attacker only has to compose them to reach unauthorized outcomes. Capability scoping is the best-known defense — and my current read is that it belongs in central config, not scattered across agent code.

Enterprise implication: AI governance becomes a config review, not an architecture review. Change config, not code.

Since v0.8.4, Control by Config extends to the OS layer — under OpenShell mode, the network policy itself is also declarative config:

openshell policy update oab \
  --add-endpoint "discord.com:443:read-write:rest:enforce" \
  --add-endpoint "chatgpt.com:443:read-write:rest:enforce"

Acceptable egress endpoints move from application config to sandbox-level network policy (OpenShell lists network policy under the dynamic policy boundary; the host can runtime-refresh it). AI governance shifts from an architecture review to a config + policy review — the same principle pushed one layer deeper.

Principle 2: Security by Architecture

No HTTP port is not a feature. It's a design constraint.

Most AI gateway products assume "I'm a SaaS, so I expose an HTTP API." That's fine for B2C apps and startup MVPs. Inside an enterprise — especially in data-sensitive industries — an HTTP port is attack surface.

Why so severe? Because AI endpoints don't fit traditional web-app threat models:

• Most WAF rules are blind to prompt injection
• AI endpoint inputs are free-form text — validation barely works
• Reach the endpoint and you can attempt session hijacking, credential stuffing, indirect prompt injection

Worse: many enterprises place their AI gateway on an "internal" segment exposed to employees, but the intranet is not safe. Lateral movement, phished employees, contractor laptops — any attacker inside the network can hit the AI gateway directly.

OpenAB's agent communication uses stdio JSON-RPC with process group isolation:

Chat Platform → OpenAB Gateway → [stdio JSON-RPC] → AI Agent
                                    ↓
                          Process Group Isolation
                          Session Destroyed After Use

Real validation: EchoLeak (CVE-2025-32711) and Salesforce ForcedLeak (CVSS 9.4) both exploited exposed HTTP surfaces. OpenAB's stdio-only design makes these attacks structurally impossible — not because of a firewall rule, but because there's no HTTP port to attack. No listener, no remote-injection path.

Concretely — openab-agent (since 0.8.4-beta.5) is a 7MB-per-session Rust native agent, 28x lighter than Pi, 55x lighter than Kiro CLI. Process isolation isn't a theoretical claim here — at 7MB per process, isolation has essentially no cost.

The choice looks radical, but the most robust path I've seen so far is simply to apply the 30-year-old "don't expose data stores to the internet" principle to AI gateways — not an invention, just pulling a known-good constraint into a new domain.

Enterprise implication: Security is not a config toggle. It's an architectural property. If your AI agent speaks HTTP, it can be attacked.

Since v0.8.4, Security by Architecture adds four more OS-level constraints:

• Process isolation — the agent runs inside its own OpenShell sandbox (unprivileged identity + seccomp; compute drivers support Docker / K8s / Podman / MicroVM).
• Filesystem isolation — the agent can't see the rest of the host filesystem (Landlock-based restriction).
• Credential injection by gateway — bot tokens / API keys are exposed to the sandbox process as placeholders by the OpenShell Gateway; the real secret is filled in by the proxy at egress request time, and the agent never sees the raw credential.
• Default-deny egress — the sandbox denies all egress by default; an allowlist opens specific destinations (configured host-side, the agent can't change it).

If an attacker prompt-injects an agent, they can only reach capabilities inside that sandbox — invisible to the host, untouchable by other sandboxes, no egress out. This isn't "hoping it's safe"; it's "sandbox boundary + policy enforcement blocking it."

Principle 3: Accountability by Default

Every action traces to a human. No exceptions.

When an AI agent writes bad code, deletes production data, or leaks a secret — who's responsible? In traditional dev, git blame answers. In AI agent dev, it's a black box.

It gets worse at scale. If you run 5 agents, 3 LLMs, and 4 third-party MCP servers, you're generating 12 independent audit streams. IT has no time to correlate them by hand. When an incident hits, reconstruction is nearly impossible — attack chains span streams, and "who did what, then what touched what" doesn't reassemble. And when SOC tries to plug into SIEM, every tool has a different log shape, every vendor a different API. Onboarding one tool is a week of integration work.

OpenAB's answer has two layers.

Layer 1: sender_context injection. Identity rides on every prompt:

sender_context = {
  user_id: "blake",
  channel: "frontend-code-review",
  timestamp: "2026-05-22T10:30:00Z",
  agent: "erwin",
  session_id: "abc-123"
}

This means:

• Every line of agent-generated code has a human on record who instructed it
• Every agent action is logged in the platform thread (Discord/Slack)
• Post-incident audit is a complete trace, not guesswork
• You can ask "who told the agent to do this?" and get a provable answer

Layer 2: centralized audit & egress. All AI traffic (agent calls, LLM requests, MCP tool invocations, egress traffic) flows through the gateway and produces a single-format audit log. SIEM plugs into one endpoint — covers everything. Egress traffic goes through the same gateway, with allowlists enforced centrally — anomalous destinations trigger alerts in real time.

The payoff shows up in incident response: what used to take days to reconstruct collapses to five minutes.

Enterprise implication: Without accountability, AI agents are liability multipliers. With it, they're auditable tools.

Principle 4: Portability by Design

Your enterprise AI strategy should not equal one vendor's roadmap.

AI models go through major version cycles every 12–18 months. GPT-4 → GPT-5. Claude 3 → Claude 4. Gemini 1.5 → 2.5. Every time an enterprise customer wants to switch models, if app-layer code is directly coupled to a vendor SDK, three things happen:

1. Rewrite every OpenAI / Anthropic call site
2. Redo prompts (each model has its own prompt habits)
3. Re-test every downstream flow (regression)

A startup might absorb this in 1–2 weeks. An enterprise typically starts at 4–8 weeks. The cost of vendor lock-in = the delay on every upgrade.

And lock-in isn't only "slower upgrades." You picked Claude today because it's the best coder. But what happens when:

• Anthropic changes terms (60+ accounts suspended overnight)?
• A better model emerges that isn't Claude-compatible?
• Your sensitive data can't leave the network?

OpenAB's ACP abstraction layer solves this:

┌─────────────────────────────────────────────┐
│  One Interface (ACP)                        │
│                                             │
│  Claude Code  Codex  Gemini  Cursor  Grok   │
│  Local LLM    GPT    Grok 4  Grok 4.20      │
└─────────────────────────────────────────────┘

Switching takes one line in config.toml. MCP servers don't change. Discord channels don't change. Your team's workflow doesn't change.

The deeper payoff: traffic split A/B testing. When a new model lands, route 5% of traffic through it, compare quality and cost, then ramp up. Without an abstraction layer, this capability simply doesn't exist.

Enterprise implication: Vendor independence is not anti-Anthropic or anti-OpenAI. It's pro-resilience. Your AI gateway should work with any backend, because you can't predict which backend you'll need next year.

The Multiplicative Effect

Individually, none of these principles is revolutionary — they're all common knowledge in their respective security subfields. Put them together and the effect is multiplicative:

• Control × Security = attackers can't reach you from outside, can't pivot from inside, and stolen credentials are short-lived and scope-bound
• Accountability × Portability = switching vendors doesn't lose your audit history; abnormal credential usage becomes immediately visible
• All four together = N AI tools collapsed into one external attack surface, behind one abstraction layer

An enterprise AI gateway controls by config, secures by architecture, accounts by default, and ports by design.

Not because each principle is nice to have. Because missing any one creates an enterprise-risk-shaped hole:

Who Should Care, and Who Shouldn't

Honestly: not every company needs all four.

You probably don't need this: B2C startups, products that are cloud-native by design, businesses where the data itself is meant to be public (social, media, ad-tech). For these, Cloudflare AI Gateway is plenty.

You do need this:

• Semiconductors, HBM/CoWoS supply chain, panels, precision machinery (process data = lifeblood)
• Banking, insurance (compliance + extremely sensitive customer data)
• Government, defense (data exfiltration is a national-security issue)
• Healthcare, pharma (clinical data, unpublished specs)
• Any enterprise whose moat is "data we own"

If you're in the second group, these four principles may be worth checking your current gateway shortlist against — a week of serious evaluation can quietly head off an incident source 12–24 months out.

Market Landscape: Why Existing Solutions Fall Short

From what I've seen so far, no product on the market does all four at the same time — at least not as a SaaS. That's the reason an abstraction layer like OpenAB exists: not because the incumbents are bad, but because their design starting point wasn't "extremely data-sensitive enterprise."

Why This Matters Today

AI is accelerating attack and defense simultaneously, but attack improvements ≈ 2× defense improvements. Google DeepMind used Gemini to autonomously find a real SQLite 0-day (CVE-2025-6965 — the first time an AI agent discovered a production-grade software vulnerability with no human guidance). Darktrace 2024 reported AI-generated phishing growth of 135% year over year.

Defensive AI automation cuts 60–80% of SOC L1 time, but that's a linear improvement. The attack-side acceleration is exponential. The traditional "buy more defensive tools" procurement logic is breaking. Strategy has to shift from "stack more defenses" to "compress attack surface."

Enterprise AI gateways are the single most effective lever for compressing attack surface. Today's design choice sets tomorrow's incident probability.

→ wchung.tw/blog/openab-series

Side note — the same team applied these four principles to the GitHub API and open-sourced ghpool (GitHub API Proxy). Useful as a reference of this pattern applied to a different surface: PAT pooling (Control), runs inside K8s private network with no external HTTP (Security), all tokens dynamically loaded from secrets manager (Accountability), works on any K8s (Portability). Same pattern, different scope.

OpenAB is an open-source ACP broker that bridges Discord/Slack/Telegram to AI coding agents. Built on these four principles.