What is an LLM gateway, and do you actually need one?
Gateway, proxy, router, aggregator -- four words for overlapping things. Here is what each one actually does, the problem they exist to solve, and the honest case for not running one.
The term gets used for four different things, which is why explanations of it tend to be vague. This is an attempt at a precise one: what the category actually contains, the specific problem it solves, how to evaluate an option, and -- the section most of these leave out -- when the right answer is not to run one at all.
The problem it exists to solve
Start with the situation rather than the tool.
You have some number of clients that want a model: your editor, a terminal agent, a script, maybe an application. Call it N.
You have some number of places a model can come from: a metered API, a subscription you already pay for, a free tier, a local model. Call it M.
Without something in between, you wire N to M directly, and you get N×M problems:
Every provider speaks a different protocol. The OpenAI Chat Completions shape, the Anthropic Messages shape, the Gemini shape, the OpenAI Responses shape, and several vendor-specific ones. A client written against one cannot talk to another.
Every provider authenticates differently. Some take an API key. Some need OAuth through a browser. Some use short-lived tokens that need refreshing. Some are subscriptions with no public API at all.
Every provider fails differently. A 429 from one means "wait 60 seconds"; from another it means "you are out until tomorrow". Handling that well requires knowing which.
Credentials multiply. Each client needs each provider's secret, so a key rotation means editing several configs.
Cost is invisible. Spend is spread across providers with no single view of what went where.
A gateway is one process that sits at the join and turns N×M into N+M. Every client speaks one protocol to it; it speaks each provider's protocol on the other side.
The four words, disambiguated
They overlap, and most products are more than one of these. Precision helps anyway.
Proxy. Forwards requests without changing their shape. Useful for adding auth, logging, or a network hop. A pure proxy does not let an OpenAI-shaped client talk to Anthropic.
Translator. Converts between API shapes. This is the part that makes heterogeneous providers interchangeable, and it is the genuinely hard component -- more on that below.
Router. Decides which provider handles a request, and what to do when one fails. Failover, load spreading, cost-based selection.
Aggregator. A hosted service that has already done all of the above and sells you access behind one key. You do not run it; you pay a margin.
"Gateway" in practice means proxy plus translator plus router, running somewhere you control. An aggregator is the hosted alternative to running one.
The five jobs a gateway does
1. Normalise the protocol. Accept one request shape, emit whatever the target needs, convert the response back. Both directions, including streaming.
2. Hold credentials. One place that knows every provider's secret, so clients hold one token for the gateway instead of nine for nine providers.
3. Route and fail over. Pick a backend, detect failure, move on. The quality of this is mostly about telling failure types apart -- a rate limit that clears in a minute needs different handling from a quota exhausted until tomorrow.
4. Meter. Count tokens and cost per request, per model, per provider. You cannot control spend you cannot see.
5. Guard. Validate what goes out. A gateway that lets a caller specify an arbitrary upstream URL is an SSRF vector pointed at your own network, so any serious one restricts that.
Why translation is the hard part
Everything else is plumbing. Translation is where implementations actually differ, and it is worth understanding because it explains why "supports 100 providers" is a weaker claim than it sounds.
The shapes disagree structurally, not cosmetically:
- Message content. One format uses a plain string; another uses an array of typed blocks. Converting loses or invents structure.
- System prompts. Some formats have a dedicated field, others expect a message with a special role, others have an
instructionskey. - Tool calls. Names, argument encodings, and the way results are returned all differ. Some formats cap the length of a call identifier, so a passthrough that ignores this produces rejected requests.
- Streaming. Different event names, different chunk shapes, different terminators. A translator has to convert a live stream without buffering the whole response, or streaming stops being streaming.
- Reasoning content. Extended thinking is represented differently everywhere, and dropping it silently changes model behaviour on formats that expect it back.
- Images. One format nests a URL object; another takes a bare string. A translator that mangles binary fields turns a working request into a 400.
None of that is visible in a feature table. It shows up as "tool calling works with provider A but not provider B", which is the practical test.
Hosted or self-hosted
The real fork in the road.
Hosted means zero operations, instant access to models you have no account with, and one invoice -- in exchange for a margin on every request and your prompts transiting infrastructure you do not run.
Self-hosted means no margin, no third party in the path, and the ability to use access you already hold -- in exchange for managing credentials and being your own on-call.
The decisive question is usually not price or privacy. It is this: do you already pay for model access a hosted service cannot reach? A Copilot subscription, a Cursor plan, a Kiro or Antigravity free tier -- these are subscription logins, not API keys, so no aggregator can use them. If you hold two or three, an aggregator bills you per token for capacity you have already bought.
If every provider you would use is a metered API key, hosted is probably the better trade and self-hosting buys you a margin you may not notice.
Individual or team
A second fork, orthogonal to the first, and the one most comparisons miss.
Team gateways need identity: virtual keys per person or service, budgets attached to them, spend attribution, and observability hooks into whatever your organisation already uses. Several people share the thing and someone has to answer for the bill.
Individual gateways need none of that and instead need the awkward auth: browser logins, subscriptions with no public API, and intercept for clients that hardcode their endpoint and offer no setting to change it.
These are different products wearing the same category label. Picking the team tool for solo use gives you configuration you never touch; picking the individual tool for a platform gives you no way to answer "who spent this".
When you do not need one
The honest section.
You use exactly one provider and one client. A gateway solves N×M. When N and M are both one, it is a hop that adds latency and a process that can stop.
You are on a subscription you never exhaust. If your allowance covers your use, routing does not save anything.
Your application is in one language with one SDK. If your code calls one vendor's SDK and you have no intention of changing, an abstraction layer is speculative.
You cannot take on the operational surface. As shared infrastructure, a gateway needs the same care as any dependency. If nobody will own that, a hosted service is more honest than a self-hosted one nobody maintains.
The case gets strong at the point where you have several clients, several providers, or model access you already pay for that your tools cannot reach. Below that, it is optional.
How to evaluate one
Feature counts are close to useless. These questions are not:
Does tool calling work end to end on the providers I actually use? Test it. This is where translation quality shows.
Does streaming stay a stream? A gateway that buffers a full response before emitting it has broken the thing you wanted.
Does it distinguish rate limits from exhausted quotas? Retrying an exhausted account every thirty minutes produces a stream of failures. Parking it until the real reset does not.
Can it authenticate the way my providers actually work? If your cheapest access is an OAuth subscription, an API-key-only design cannot reach it.
Where do my prompts go? A local process and a hosted service are different answers, and only one of them is a policy question.
What happens when it is down? Know this before it happens rather than during.
Running one
For the individual case:
npm install -g @sifxprime/krouter
krouter -tConnect providers in the dashboard, then point clients at one endpoint:
export OPENAI_BASE_URL=http://localhost:20128/v1
export OPENAI_API_KEY=sk-krouter-localClients speaking the Anthropic Messages API use their own variables instead -- Claude Code appends /v1/messages itself, so it takes the root:
export ANTHROPIC_BASE_URL=http://localhost:20128
export ANTHROPIC_AUTH_TOKEN=sk-krouter-localVerify before trusting it:
curl -s http://localhost:20128/v1/chat/completions \
-H "content-type: application/json" \
-H "authorization: Bearer sk-krouter-local" \
-d '{"model":"gh/claude-haiku-4.5","messages":[{"role":"user","content":"ready?"}],"max_tokens":16}'A choices array means the whole path works. A 401 means a provider needs reconnecting.
Common questions
What is the difference between an LLM gateway and an API gateway?
An API gateway routes HTTP by path and method and does not understand the payload. An LLM gateway understands the payload -- it converts between model API shapes, counts tokens, and makes routing decisions based on model and quota state. The general-purpose one cannot let an OpenAI-shaped client talk to Anthropic; that conversion is the specific thing an LLM gateway adds.
Is an LLM gateway the same as a proxy?
A proxy forwards requests unchanged. A gateway also translates between formats and decides which provider handles each request. Every gateway contains a proxy; not every proxy is a gateway.
Does a gateway add latency?
A local one adds roughly a millisecond plus translation, which is not perceptible against a multi-second model response. A hosted one adds a network hop to their infrastructure. Neither is usually the thing worth optimising.
Do I need one if I only use one provider?
Probably not. The value comes from having several clients or several providers. With one of each it is a hop that can fail.
Can a gateway use my Copilot or Cursor subscription?
Only one running locally that speaks their protocol. Those are subscription logins rather than API keys, so a hosted aggregator has no way to authenticate as you.
Is self-hosting more secure?
Different, not automatically better. Your prompts stay on your machine, which removes a third party. But you are now responsible for the credentials and for not exposing the endpoint. A gateway reachable from your network with a weak token is worse than a reputable hosted service.
Related
Klaw is the Kodelyth AI agent. He writes drafts, runs the benchmarks, and tracks every cost number in this post live through kRouter. Humans review before publish.
Install kRouter