Building a sub-5ms failover proxy: moving from SQLite to RAM

When a local AI proxy fails over from one account to the next, the network is not the only cost. Before v0.5.69, kRouter paid for disk too.

When a provider returned a 429, kRouter wrote a lock for that account to SQLite, synchronously, and then the routing loop read SQLite again to find the next available account. Two trips to the database per dead account. If several accounts were exhausted in a row, the request waited on all of those reads and writes before it went anywhere useful.

v0.5.69 moved that state into memory. This post explains the change.

What changed in this post

The first version of this post had a table of before-and-after latencies for 1, 5, 15 and 20 dead accounts, a batched write queue that flushed every 500 ms, and a list of state hydrated at boot. The table was not from a benchmark in kRouter's repository, and the write queue and boot list did not match the code. The release notes for v0.5.69 say failover overhead was about 50 ms per failure before the change and under 1 ms after; those are the project's own figures, not a published benchmark, and the "sub-5ms" in the title is that release's stated goal. What follows is how the code works.

The health cache

kRouter keeps an in-memory copy of every active provider connection, grouped by provider, including each connection's locks. It loads that copy from SQLite and refreshes it whenever it is more than 10 seconds old.

When a request needs an account, kRouter filters the in-memory list: it drops accounts already tried for this request, accounts flagged as permanently banned, and accounts locked for the requested model. The Zenith engine then scores what is left and picks one. None of that touches the disk.

The hot path after a 429

1. The provider returns a 429, or another error that should take the account out of rotation.
2. kRouter applies the lock to the in-memory record immediately, so the next pick skips that account.
3. It starts the SQLite write in the background and does not wait for it. If the write fails, the failure is logged.
4. The next account is picked from memory and the request is retried there.

The database still gets every lock; it just stops being in the way.

What lives where

Two kinds of state, two homes:

• Account locks and errors go to SQLite as well as memory, so a long lock -- a daily quota that resets in hours, an account flagged after a ban signal -- survives a restart.
• Health scores (recent success rate and request time, which Zenith uses) are memory only. They are a short-term view anyway: kRouter rebuilds them from live traffic within minutes of starting, and new accounts start from a neutral score.

SQLite runs in write-ahead logging mode, so the background writes do not block the reads that refresh the cache or serve the dashboard.

What can go wrong

The background write is fire and forget. If kRouter exits before a lock is written, the account looks available after the restart, gets picked, fails once, and is locked again. That costs one wasted request, not a stuck account.

A cache refresh replaces the in-memory copy with what is in the database, so a lock whose write has not finished yet can be missing for a moment. Again, the cost is at most one retry against that account.

Why keep SQLite at all

Because machines restart. If your laptop sleeps or you restart krouter, it needs to remember that an account is locked until tomorrow's reset. Memory is for speed; the database is for anything that has to outlast the process.

Get it

The health cache has been in kRouter since v0.5.69, and nothing needs configuring. Any current install has it:

npm install -g @sifxprime/krouter@latest

Related

• The Zenith routing engine: how kRouter picks an account
• The Zenith routing engine docs
• Self-host an AI router on a VPS