Getting duplicate AI replies? The response-cache gotcha, explained

You say "Hi" to your AI assistant. It replies, "Hello, how can I help you?" Then you ask something completely different — and it replies "Hello, how can I help you?" again. The same canned answer, for the wrong question.

If you have a response cache turned on, that cache is almost always the culprit. This post explains the exact failure mode, why it hits IDE-driven providers like Antigravity hardest, and how kRouter's cache guards prevent it.

Why a response cache exists

Many IDE and agent workflows fire the same request over and over within seconds — warmup pings, title generation, "is this model reachable" probes, structured-output retries. Serving those from a short-lived in-memory cache saves real provider tokens with no downside, because the request is byte-for-byte identical.

A response cache keys each entry by the request shape: model, system prompt, messages, temperature, max tokens, tools. Same shape in, same response out.

Where it goes wrong

The trouble starts with providers whose IDE fires small deterministic probe requests at temperature 0. Antigravity's Google backend is the classic example. Those probes have nearly identical bodies, so they collapse to the same cache key. Then a real user turn arrives, hashes to that same key, and gets served the probe's canned reply.

That's the "Hello, how can I help you?" bug. The cache is doing exactly what it was told — it just can't tell a throwaway probe from a real conversation turn, because on the wire they look the same.

The fix: cache guards

kRouter caches aggressively where it's safe and refuses to cache where it isn't. The guards, in order:

1. Provider blocklist. Providers with backend session state and heavy IDE probe traffic — Antigravity, Gemini, Gemini CLI — never get cached, regardless of your global toggle. This alone makes the "Hi" bug impossible on those providers.
2. Streaming skip. Streaming responses are never cached — replaying SSE chunks correctly across providers is a different problem.
3. Temperature gate. Anything above temperature 0.3 is treated as non-deterministic and skipped.
4. Probe-size skip. Requests with max_tokens under 32 are almost always IDE warmup pings, not real turns worth a cache hit later.
5. Empty-reply skip. Responses under 100 bytes — error stubs, empty completions — are refused so they can't poison the cache.

There was a subtle trap in getting this right, worth calling out because it's a lesson in how routers actually work: the blocklist has to match the provider alias that appears in the model string, not just the provider id. An Antigravity request arrives as ag/gemini-3-flash-agent, not antigravity/.... An early version of the guard only listed the ids, so it never fired. The blocklist now covers both forms — antigravity and ag, gemini-cli and gc — verified live against a running instance.

What you should do

• Leave the response cache off if you're unsure — it's off by default.
• Turn it on if you route a lot of repetitive deterministic traffic (batch jobs, structured extraction, repeated title generation) on providers other than the Google family. You'll save real tokens with no wrong-answer risk.
• On Antigravity/Gemini, turn it on freely — kRouter simply bypasses the cache for them, so you get correct answers whether the toggle is on or off.

Try it

npx @sifxprime/krouter@latest

The cache toggle lives on the Endpoint page. See the features overview for everything else the router does to keep responses correct and cheap.