The LLM Gateway: Why You Stop Calling Model APIs Directly in Production

The pattern shows up in every LLM codebase that grew from a prototype: client = OpenAI() imported in forty files, the API key in each service's environment, the model name hardcoded three levels deep. It works beautifully right up until the afternoon the provider has a partial outage, and now you're grepping forty files to swap in a fallback, redeploying under pressure, and discovering that half of them handled errors slightly differently. I've lived that afternoon. It's the LLM-era rerun of a lesson the API world learned a decade ago and solved with a gateway — and the fix is the same.

An LLM gateway is that solution ported to model traffic: a single proxy that every service calls instead of calling providers directly, which then handles routing, failover, keys, budgets, caching, and guardrails as centralized policy rather than scattered code. If you've used an API gateway like Kong or Apigee in front of microservices, you already understand the shape — this is the same architectural move, applied to the new kind of backend that happens to be an LLM. This piece is about what that layer actually buys you, where it sits, and how the main options — LiteLLM, Portkey, Kong AI Gateway — genuinely differ, because they're less interchangeable than the "AI gateway" label suggests.

What is an LLM gateway, and what does it actually do?

An LLM gateway is a proxy that sits between your applications and every model provider, exposing one unified (usually OpenAI-compatible) API while centralizing the cross-cutting concerns of production LLM traffic. Your code makes one kind of call to one endpoint; the gateway decides which provider and model actually serves it, and applies policy on the way through. The single most important consequence is decoupling: your application no longer knows or cares whether a request lands on GPT-class, Claude-class, Gemini-class, or a self-hosted open model, because the gateway owns that mapping.

Concretely, a mature gateway consolidates six jobs that otherwise sprawl across your codebase:

Notice how many of those are things I've written about as standalone problems — semantic caching, LLM observability, token-cost FinOps, prompt-injection guardrails. The gateway's real pitch is that it's the natural home for all of them at once. Rather than bolting each concern onto every service, you enforce them in one place that all traffic already flows through.

Where does it sit, and why there?

The gateway sits on the request path between your applications and the providers, which is precisely why it can enforce policy — everything already passes through it, so nothing has to opt in. That position is the entire source of its leverage, and it's worth seeing the flow to understand what becomes possible.

flowchart LR
    APPS["Apps · agents · services
one OpenAI-compatible call"] --> GW
    subgraph GW["LLM gateway"]
        direction TB
        AUTH["Virtual key · budget · rate limit"]
        CACHE["Cache lookup"]
        ROUTE["Route by cost / latency"]
        GUARD["Guardrails · PII redaction"]
        AUTH --> CACHE --> ROUTE --> GUARD
    end
    GW -->|"primary"| P1["Provider A"]
    GW -.->|"fallback on error"| P2["Provider B"]
    GW -.->|"fallback"| P3["Self-hosted model"]
    GW --> OBS["Unified logs · traces · cost per team"]
          

Fallback deserves the spotlight because it's the capability that turns a convenience into a reliability tool. With a fallback chain configured, a provider outage stops being an incident: the gateway catches the error or timeout, transparently retries the next provider in the chain, and your users see a slightly slower response instead of a failure. The forty-files-to-edit afternoon becomes a config entry you set once. That single property is why I now treat a gateway as closer to mandatory than optional for anything customer-facing.

How do LiteLLM, Portkey, and Kong actually differ?

They cluster into three genuinely different bets — open-source-and-self-hosted, feature-complete-SaaS, and enterprise-mesh — and picking well is mostly about which of those matches your constraints. The "AI gateway" label flattens real differences in operating model, so here's how I'd separate them.

My rough decision rule, stated plainly: pick LiteLLM if you run open models alongside cloud APIs and want an open-source proxy you control, with virtual-key budgeting that just works. Pick Portkey if you're cloud-only and want semantic caching, guardrails, and clean observability without operating anything yourself. Pick Kong AI Gateway if you already run a Kong API mesh and your requirement is enterprise governance — SSO, PII-redaction plugins — folded into infrastructure you already manage. Cloudflare AI Gateway and Helicone are worth a look too if edge-proximity or observability-first is your dominant need, respectively; the space is not just these three, but these three define the axes.

What does it look like to actually use one?

The whole point is that your application code gets simpler, not more complex — it makes one ordinary OpenAI-style call, and all the routing and fallback logic lives in gateway config, not in the request. Here's the shape with a self-hosted LiteLLM proxy: the app is trivial, and the intelligence is declarative.

# Application code: one endpoint, one call. It has no idea which
# provider serves this, and that's the entire point.
from openai import OpenAI

client = OpenAI(base_url="http://llm-gateway.internal:4000", api_key=VIRTUAL_KEY)
resp = client.chat.completions.create(
    model="chat-default",          # a gateway alias, not a provider model
    messages=[{"role": "user", "content": prompt}],
)

# Gateway config (LiteLLM): the alias "chat-default" maps to a primary
# model with a fallback chain. Swapping providers is a config change,
# never a code change — and the app redeploys never.
model_list:
  - model_name: chat-default
    litellm_params:
      model: anthropic/claude-primary
  - model_name: chat-default          # same alias, second option
    litellm_params:
      model: openai/gpt-fallback

router_settings:
  fallbacks: [{ "chat-default": ["chat-default"] }]  # try the next entry on failure
  routing_strategy: latency-based-routing

That model_name alias is the hinge of the whole pattern. The application asks for a capability ("give me the default chat model"), not a vendor, and the gateway resolves that to a concrete provider plus its fallbacks. Shifting 20% of traffic to a cheaper model, adding a new provider, or reordering the fallback chain during an incident are all config edits to one system — none of them touch application code, and none of them require the forty-file grep.

What's the catch?

A gateway is a single point that every LLM request now depends on, and pretending otherwise is how it becomes the outage instead of preventing one. Treat it as the critical infrastructure it is.

Two smaller cautions worth holding. Self-hosted options (LiteLLM, Kong) are yours to keep highly available and patched — the ops you removed from application teams didn't vanish, it moved to whoever runs the gateway. And the managed SaaS options (Portkey) mean your prompts and completions transit a third party, which is a data-governance conversation to have deliberately, not a checkbox to tick past — for regulated workloads it may be the deciding constraint, pushing you to a self-hosted gateway regardless of feature comparisons.

What to carry away

The LLM gateway is the API-gateway pattern arriving, on schedule, for model traffic — and the forcing function is the same one that created API gateways: cross-cutting concerns (auth, cost, caching, failover, safety) that are miserable to duplicate across every service and clean to enforce in one place the traffic already flows through. The capability that justifies it on its own is automatic fallback, which quietly converts a provider outage from an all-hands incident into a config line. Choose the implementation by operating model, not feature checklist — LiteLLM to self-host with control, Portkey for managed caching-and-guardrails, Kong to fold into an existing mesh — and respect that you've centralized your reliability into one component, so run it like it matters. Get that right and your application code stops knowing which vendor it talks to, which is exactly the freedom you want in a market where the best model per dollar changes every few months.