Limits, retries and refusals

Rate limits come in two shapes, retries need idempotency to be safe, and a refusal can arrive as a perfectly successful response.

Part of the Cheap and fast track on lAItest.

The failure you have not planned for is the one that returns HTTP 200.

Limits come in at least two shapes

Providers meter requests per minute and tokens per minute separately, and you can sit comfortably inside one while hitting the other. A flood of tiny calls trips the request limit. One enormous document trips the token limit. Both come back as a rejection with a hint about when to try again, and the right response is to wait a growing amount of time, with some randomness added so your whole fleet does not retry in the same instant.

Retrying safely is the hard half

A retry is only safe if the server can tell it is a repeat. That is what an idempotency key is for: the same key on the same request means do the work once and return the same answer twice. Without one, a timeout that was never your fault becomes a duplicate charge, a duplicate email, or two rows where you wanted one. The request that timed out may well have succeeded.

A common misconception

Commonly believed: If the call came back 200, the model answered.

Actually: Not always. Some current frontier models can decline a request, and a decline arrives as a successful response carrying a refusal stop reason and a category, not as an error. Code that reaches straight for the first content block breaks on it. The billing is unusual too: a refusal issued before any output is not billed at all, while one that arrives mid-stream is billed for what already streamed.

A streaming speech API bills per session. Your app opens the socket on page load and closes it on exit. What are you paying for?

Answer: Every second the socket is open, silence included. At least one major vendor bills streaming transcription by how long the connection stays open, not by how much audio crossed it. Idle time is billable time. Read what the meter counts before you read the rate.

What this lesson was checked against

In one sentence

Assume every call can be rejected, repeated, or politely declined, and make all three of those cheap to handle.