Best self-hosted LLM gateways in 2026, and how to vet one

On March 24, 2026, for about 40 minutes from 10:39 UTC, an unpinned pip install litellm fetched version 1.82.7 or 1.82.8. Both carried a payload that collected environment variables, SSH keys, cloud credentials, Kubernetes tokens and database passwords, encrypted them, and posted them to models.litellm.cloud, a domain LiteLLM does not own. PyPI quarantined the releases, LiteLLM shipped a clean 1.83.0 through a new CI/CD pipeline, and its incident report tells anyone who installed them to rotate every secret on that machine.

None of that is specific to LiteLLM. A gateway is, by design, the process that holds every provider key you own. So two questions matter more than the benchmarks and provider counts most roundups rank by: who is it built for, and can you check what you are running?

This post answers both for the self-hosted options worth considering in 2026, including where kRouter, the one we make, is the wrong pick.

Team gateway or personal router

The category holds two different products.

A team gateway sits in front of many people and services. It needs identity: virtual keys, budgets attached to them, spend attribution and observability exports. That means a database, and someone to operate it.

A personal router sits in front of one developer's tools. It needs none of the accounting, and instead needs sign-ins to subscriptions you already pay for, several accounts per provider, and a way to reach clients that hardcode their endpoint.

If someone has to answer "who spent this?", you want the first kind.

The candidates

LiteLLM. The default for teams. A Python SDK plus a proxy server; virtual keys, per-key budgets and spend tracking need Postgres. A Rust core for provider translation is in beta, off unless you enable it per model. It can also sign in to GitHub Copilot and a ChatGPT subscription with a device code. MIT, except an enterprise/ directory under its own license.

Bifrost. A Go gateway from Maxim AI, under Apache 2.0. npx -y @maximhq/bifrost or the Docker image starts it with a web UI on port 8080, keeping configuration in SQLite by default, with Postgres supported. Virtual keys and budgets, fallbacks and semantic caching are in the open-source build, on a single node; clustering, adaptive load balancing and guardrails are enterprise features. Its speed claims against LiteLLM come from its own benchmarks.

Portkey Gateway. A TypeScript gateway under MIT, known for routing configs and built-in guardrails. On March 24, Portkey said its full gateway, governance and cost controls included, was now open source. On May 29, Palo Alto Networks closed its acquisition of Portkey, which becomes the core AI gateway of Prisma AIRS. The public repository's last tagged release is 1.15.2, from January, and its last commit is from May. Check where development continues before you build on it.

LLM Gateway. A TypeScript project that runs hosted at llmgateway.io or on your own hardware, where one Docker image bundles Postgres and Redis. The core is AGPLv3 and an ee/ directory is commercial. AGPL obligations apply if you modify the code and let others use it over a network.

Agent Router. Envoy AI Gateway was renamed Agent Router in September 2026 and is now an Agentic AI Foundation project; the code, Apache 2.0 license, container images and aigw CLI are unchanged. It is built for Kubernetes on Envoy Gateway, and aigw run starts a standalone router on a laptop. The obvious choice if your platform team already runs Envoy.

kRouter. A Node application (20.9 or newer) under MIT, installed from npm or Docker, keeping everything in SQLite, with a dashboard on port 20128. It is built for one developer: it signs in to subscriptions such as Claude Code, Codex, GitHub Copilot, Kiro and Antigravity, takes API keys for the rest of its 95+ providers, rotates several accounts per provider, falls back through combos, and translates between OpenAI, Anthropic and Gemini formats. It can also intercept Claude Desktop, Antigravity, Copilot and Kiro IDE traffic locally. It issues API keys and shows usage and estimated cost per key, but has no users, teams, budgets or spend limits. Before you connect a subscription, the dashboard warns that a provider can restrict an account it sees used through a proxy. The comparison page sets it against 9router, the project it forked from, and others.

TensorZero, still in older lists, is archived; its last release was in June 2026.

Side by side

What the March compromise teaches

Installs pinned to an earlier version were not affected. The official LiteLLM proxy Docker image pins its dependencies and was not hit. The exposure came from installs that took whatever was newest during those 40 minutes.

The payload wanted the machine, not the gateway. It went after SSH keys, cloud credentials and Kubernetes tokens as well as model keys. A host that runs a gateway should hold nothing else worth stealing.

Incident handling is part of the product. LiteLLM published the versions, the window and what to rotate, which tells you more than a feature list.

Vetting a gateway before it holds your keys

All of this applies to kRouter too.

1. Pin an exact version. Quickstart one-liners (:latest, an unversioned npx) are for trying a tool, not for the machine that holds your keys. Update on purpose, after reading the release notes.

pip install "litellm==1.104.0"
npm install -g @sifxprime/krouter@0.5.163
docker pull sifxprime/krouter:0.5.163

2. Check how the release was built. npm and PyPI can attach provenance, a signed record of the repository, tag and workflow that produced a package. On npm you can look before installing:

npm view @sifxprime/krouter@0.5.163 dist.attestations

For kRouter this prints an SLSA provenance entry, which traces the release to the sifxprime/krouter repository, its version tag and the publish workflow. Empty output means there is none. Inside a project, npm audit signatures checks the registry signatures and provenance of everything installed.

Containers are separate. LiteLLM signs its GHCR images with cosign from 1.83.0 on; its Docker image security guide gives the cosign verify command. Use that path, because PyPI attestations appear on its 1.83 releases but not on later ones such as 1.84.0 and 1.104.0. kRouter's Docker images are not signed and carry no provenance; if that matters, install from npm or build the image from a tagged clone.

3. Check who can reach it. kRouter binds to 0.0.0.0 by default, so it is reachable from your network, and it says so at start. Bind it to loopback unless something else needs it:

krouter -t --host 127.0.0.1

One catch: start-on-login, whether you turn it on with Hide to Tray + Start on Login in the terminal menu or Enable Auto-start on the tray icon, launches kRouter without --host, so after a reboot it listens on every address again. On Windows and Linux, that menu option also restarts kRouter in the background without --host straight away. lsof -nP -iTCP:20128 -sTCP:LISTEN on macOS, or ss -ltn on Linux, shows which address it holds.

Local callers need no key; network callers need one from the dashboard's Endpoint page, and REQUIRE_API_KEY=true removes the local exemption. Under Docker, requests from your host reach the container from outside it, so clients need a key there too. Publish the port on loopback, pin the tag, and generate the dashboard password:

KROUTER_PASSWORD="$(openssl rand -base64 18)" && echo "Dashboard password: $KROUTER_PASSWORD"
docker run -d -p 127.0.0.1:20128:20128 \
  -e INITIAL_PASSWORD="${KROUTER_PASSWORD:?run the line above first}" \
  -v "$HOME/.krouter:/app/data" --name krouter sifxprime/krouter:0.5.163

4. Check what it keeps on disk. kRouter keeps OAuth tokens and API keys in a SQLite database in its data directory (~/.krouter by default), not encrypted at rest, so treat that directory and its backups like ~/.ssh. If you turn on MITM mode, the same directory holds the private key of the root certificate it installs; anyone who copies that key can impersonate any website to that machine while the certificate is trusted. kRouter also records the latest 1,000 requests and responses for its dashboard by default, small ones whole and larger ones as a 200-character preview; turn off Enable Observability in Settings if you do not want prompts on disk.

5. Check whether it updates itself. A tool that installs new code on its own has quietly unpinned itself. kRouter's CLI and dashboard tell you when npm has a newer version and show the install command, but neither installs it on its own; --skip-update turns off the CLI's check.

6. Read its security history. You want a project that writes vulnerabilities down when it fixes them, and says which ones it left open. kRouter's security docs list the releases that fixed vulnerabilities, such as a remote auth bypass through a spoofed Host header (0.5.135) and a Next.js flaw allowing remote code execution on Windows hosts (0.5.157). Its changelog also names dependency advisories that could not be reached or were left open on purpose, and why. The self-hosting checklist covers the rest.

7. Check that it is maintained, and by whom. Of the projects in this post, one was acquired this year, one archived and one renamed. Keep clients on a standard OpenAI- or Anthropic-compatible base URL, so changing gateways is a configuration change, not a migration.

Choosing

kRouter is the wrong pick for team budgets and per-user spend limits, for cluster infrastructure, and under a policy that requires signed container images, which its Docker images are not. If you would rather run nothing, the OpenRouter comparison covers when a hosted aggregator beats self-hosting. kRouter fits when you are routing your own work and model access you already pay for.

Common questions

What is the best self-hosted LLM gateway in 2026?

It depends on who it serves. For a team that needs virtual keys, budgets and spend attribution, look at LiteLLM or Bifrost, or Agent Router on Kubernetes. For one developer routing their own subscriptions and API keys, a personal router such as kRouter fits better.

Is LiteLLM safe to use after the March 2026 compromise?

The malicious versions were 1.82.7 and 1.82.8 on PyPI, live for about 40 minutes from 10:39 UTC on March 24, 2026. The official proxy Docker image was not affected, and 1.83.0 was clean. Since 1.83.0 its GHCR images are signed with cosign, so pin a version and verify the image. LiteLLM's report asks anyone who installed or upgraded it with pip between 10:39 and 16:00 UTC that day to rotate every credential on that machine and check for a litellm_init.pth file.

How do I check whether an npm package has provenance?

Run npm view <package>@<version> dist.attestations before installing. A provenance entry traces the release to the repository, tag and workflow that built it; empty output means there is none. In a project, npm audit signatures checks everything installed.

Does self-hosting a gateway keep my prompts private?

Only partly. Prompts still go to whichever provider answers them, and the gateway may store them. kRouter keeps recent requests and responses for its dashboard by default; turn off Enable Observability in Settings if you do not want them on disk.

Related

• What is an LLM gateway, and do you actually need one?
• LiteLLM vs kRouter: they solve different problems
• OpenRouter alternative: when self-hosting is worth it
• The security checklist for self-hosting an AI router
• kRouter compared with 9router, OmniRoute, LiteLLM and OpenRouter