Skip to main content
kRouter
All posts
Fix an error

A 200 that is a retirement notice is not an answer

A provider replying "that model is gone, use this other one" has every marker of success: HTTP 200, finish_reason stop, real text. It was billed, logged as fine, and shown to the user as the model's own words.

Kodelyth · The team behind kRouter
8 min read

Here is a response that is entirely successful by every test a gateway normally applies:

HTTP 200
{"object":"chat.completion","model":"gemini","choices":[{"message":{
  "content":"Claude Sonnet 4.6 is no longer available. Please switch to Claude Sonnet 5.5."},
  "finish_reason":"stop"}]}

Status 200. A finish_reason of stop, which is the clean one. A message.content that is real, non-empty text. No error field, no refusal, no truncation.

It is also not an answer. It is the vendor telling you the model you asked for has been retired, formatted exactly like the model answering you.

This was measured live against ag/claude-sonnet-4-6 — an id still listed in the catalogue at the time. Every layer of kRouter looked at it and saw a success:

  • it was billed, because a 200 with content is a completion
  • it was written to the request log as 200 OK
  • it was rendered to the user as though the model had said it
  • and a combo did not advance to its next entry

That last one is the real damage. A combo exists to fall through to the next provider when one fails. Here the first entry never failed — it returned a tidy 200 — so the fallback never fired, and the user got a notice about a retired model instead of the answer the second entry would have given them.

The existing guard could not catch it. isZeroContentResponse rejects responses whose finish reason is in_progress or incomplete. A retirement notice is neither. It finished, and it has content. By that definition it is a completion.

Why this is harder than it looks

The obvious fix is to look for the words. Scan the content for "no longer available" and treat it as a failure.

That fix is worse than the bug, because of a case that looks almost identical:

User: Is Claude Sonnet 4.6 still available?

Model: Claude Sonnet 4.6 is no longer available. You should switch to Claude Sonnet 5.5.

That is a model correctly answering a question. Discarding it, billing it as a failure, and silently advancing a combo to a different provider would be a far nastier bug than the one being fixed — because now the gateway is eating legitimate answers, and the user has no way to see why.

Any detector here has to separate "the vendor is telling you the model is gone" from "the model is telling you a model is gone," and the text can be word-for-word the same.

The shape of the gate

So the test is a conjunction. Every part must hold, and any one of them failing means the response is passed through as the answer it probably is:

  • No tool calls. A response that also called a tool is doing work; it is not a notice.
  • The message is short. A vendor notice is one or two sentences. An answer about model availability has context, caveats, a recommendation.
  • It says the thing is gone and names a replacement. Notices always do both — the whole point is to route you somewhere. A passing mention of a deprecation usually does not.
  • When the caller supplied the requested model, the text must name that model.

The last condition does the heaviest lifting, and it is worth seeing why.

A vendor notice is always about the model you just asked for. That is structurally guaranteed: it is the provider's response to your request for that specific id. So if you asked for claude-sonnet-4-6 and the text names claude-sonnet-4-6, that is consistent with a notice.

A user asking about some other retired model does not match. Ask Gemini about Sonnet 4.6 and the reply names Sonnet 4.6, but the model you requested was Gemini — the names disagree, the condition fails, and the answer passes through untouched.

It is not airtight. A user who asks the exact model they are talking to whether it has been retired, and gets a short answer naming a successor, will trip it. That is a narrow enough case to accept in exchange for combos that actually fall through.

The silence underneath

The reason this went unnoticed for so long was not subtle, and it was not really about retirement notices at all.

There is a constant named EMPTY_COMPLETION_STATUS. It exists precisely so that empty completions are not filed alongside real successes in the request log. It was called from nowhere.

So every empty completion — and every retirement notice — was written to the request log as 200 OK. If you went looking for evidence that something was wrong, the log told you everything was fine. The one instrument that would have shown the pattern had been built and never wired up.

A third bug compounded it: one handler wrote its request-log line before computing its diagnosis. The log said 200 OK while the detailed record for the same request correctly said empty. Two records of one call, disagreeing, because of statement order.

That is the thing worth taking away. The detection logic above is a fiddly conjunction tuned against a specific vendor behaviour, and it will need maintenance. The underlying failure is more general and more boring: a category of failure that is recorded as a success is invisible, and no amount of looking at the dashboard will reveal it. The fix was not just catching the notice — it was making sure that when one slips through in future, the log says something other than OK.

What changed

Verified end to end on a live account: a combo whose first entry returns the notice used to return the notice to the user. It now advances to the next entry and answers from there, filing the first attempt as empty rather than success.

Twenty tests pin both directions — what must be caught, and what must be left alone. The second set matters more than the first.

If you use combos, or any model a vendor has retired, this landed in 0.5.172.

Kodelyth · The team behind kRouter

Published by Kodelyth, the team that builds kRouter. Posts are drafted with AI assistance and reviewed by a person before they go out. kRouter is free and MIT licensed.

Install kRouter