LLM API Rate Limits: 13 Providers Compared, 2 SDKs Never Retry a 429

Four of the thirteen LLM APIs surveyed here (ten vendors plus Bedrock, Vertex AI and Azure OpenAI) tell you your remaining budget on every response, Mistral sends a single remaining count, and eight say nothing until the request fails. “429” is four different situations: a throttle to retry, an acceleration limit to slow down for, a quota or spend cap not to retry at all, and an overload reported as 429, 503, or 529 depending on the vendor. And the client decides more than the vendor: the OpenAI, Anthropic and Groq SDKs retry a 429 twice and give up on any Retry-After above 60 or 120 seconds, while the Google GenAI and Mistral SDKs ship with retries off. This is the cross-vendor reference: dimensions, which tokens count, headers, 429 semantics, and what each official SDK does, read from its source.

TL;DR

• LLM APIs throttle on requests and tokens per minute; Anthropic splits input and output, DeepSeek caps concurrency only.
• OpenAI, Azure and Bedrock debit max_tokens up front; Anthropic exempts cache reads; xAI counts reasoning; Bedrock burns 10 quota tokens per Claude 5 output token.
• OpenAI, Anthropic, Groq and Azure return remaining and reset headers on every 200; eight APIs document none.
• google-genai and Mistral SDKs do not retry a 429 by default; OpenAI, Anthropic and Groq retry twice and cap Retry-After at 60 or 120 seconds.

What do RPM, TPM and the other limits mean?

They are all ceilings on how much you may send in a window, and each provider picks its own set:

The number on the page is a ceiling for the window, not an allowance: Anthropic notes that 60 RPM “might be enforced as 1 request per second”, Azure that “a burst within a 1-second or 10-second window can trigger a 429 even if the per-minute total is within limits”.

What do LLM APIs rate-limit on?

Requests and tokens per minute almost everywhere; the three clouds do not simply relay their vendors’ rules. Definitions are the providers’ own, from the linked pages as they read on 2026-09-03 and 2026-09-04.

Two rows are not minute windows at all. DeepSeek caps open connections and, under load, keeps a request open with empty lines (or : keep-alive comments on a stream) for up to 10 minutes rather than rejecting it. Vertex AI on pay-as-you-go has no quota number to read: a 429 means the shared pool was short at that moment.

Which tokens count toward the limit?

Not the same tokens you are billed for, and the difference decides whether your real capacity is a fraction or a multiple of the number on the page:

The same request is a different size against every limit, 800 to 6,000 tokens in the example above, so a router that spreads one workload across vendors needs a meter per vendor.

Which headers does each provider return?

Four APIs put the limit state in every response, Mistral sends one field, and the other eight leave you to count.

Three formats for one field is the practical problem: OpenAI’s duration, Anthropic’s timestamp and Groq’s 2m59.56s need three parsers, which is one reason the SDKs read only retry-after. Two intermediaries were measured on 2026-09-03 with one request each. A large multi-provider aggregator returned no rate-limit header at all on a 200, reserving them for 429 responses; the Synthorai gateway returned x-ratelimit-remaining-requests and x-ratelimit-reset-requests alongside the request’s cost.

What does a 429 mean, and should you retry?

It depends on which of four things it is, and only two of them deserve a retry:

The trap is in the third row. Anthropic’s spend-cap 429 carries the same rate_limit_error type as a throttle, and the docs say “Retrying, including the SDKs’ automatic retries, fails until access resumes.” OpenAI’s insufficient_quota is a 429 too. A client that branches on the status code alone retries both, and every official SDK does, as the next section shows. Branch on the error code, and treat a 429 without Retry-After as a hint that waiting will not help.

The fourth row is where the clouds differ. Azure’s troubleshooting guide says it plainly: “Many customers misinterpret capacity-related 429s as quota problems, leading to incorrect remediation.” On Azure, a 429 with x-ratelimit-limit-tokens lower than your configured limit is the shared pool protecting itself; on Vertex AI every pay-as-you-go 429 is that; on Bedrock the same condition is a 503 or 529, never a ThrottlingException. Through an aggregator it shows up in the body: one 429 during the header measurement arrived wrapped as provider_error_code: insufficient_quota, limit_source: upstream_provider_shared_pool. The status said throttle; the body said someone else’s quota.

What does your SDK do with a 429?

It depends on the SDK, and two of the most used ones do nothing. Read from each official client’s own source on 2026-09-04, not from its documentation:

Three things follow. First, a 429 from Gemini, Vertex AI or Mistral reaches your code on the first occurrence unless you passed retry options; the google-genai source comment that the client “will retry 4 times” describes the enabled configuration, not the default. Second, the Retry-After ceilings work against the long waits a vendor actually asks for: Anthropic’s SDK, asked to wait 90 seconds, ignores the header and comes back in under 8; OpenAI’s, asked to wait 150, stops retrying. Third, the SDKs generated by Stainless (the code generator behind the OpenAI, Anthropic and Groq clients) obey an undocumented x-should-retry header before they look at the status code, and retry-after-ms before retry-after, so a vendor or a gateway can steer retries without changing a status code.

None of them distinguish a spend-cap 429 from a throttle. Two retries of an insufficient_quota response cost a few seconds each, but a fleet doing that at scale is a self-inflicted burst.

How does a gateway change the picture?

A gateway is the one place where all of this can be handled once: upstream it reads whichever header each vendor sends, so the three reset formats and the four 429 meanings are its problem rather than every client’s; downstream it presents one header set and one error shape, which is what the measured Synthorai headers above are. What a gateway cannot do is change vendor accounting: behind OpenAI a client’s max_tokens still debits TPM up front, behind Bedrock a Claude output token still burns ten. And a gateway that pools many clients onto one vendor key reaches that vendor’s acceleration limit sooner than any single client would, which is the case for per-key buckets rather than one shared counter.

FAQ

Does max_tokens count toward the rate limit?

On OpenAI, Azure OpenAI and Bedrock, yes: max_tokens is debited when the request arrives, so an oversized value wastes TPM even if the answer is short; Bedrock corrects the deduction when the response finishes. On Anthropic, no: OTPM counts the tokens actually generated and max_tokens “does not factor” in.

Do cached tokens count toward the rate limit?

Not on Anthropic, Bedrock or Azure PTU deployments, where cache reads are excluded. On xAI they count in full. OpenAI and Gemini do not say.

Is Retry-After always sent with a 429?

No. OpenAI, Azure (as retry-after-ms) and Groq send it on throttles, Anthropic on throttles but not on the spend cap, Kimi and Bedrock only on overload. Gemini, Vertex AI, xAI, Alibaba, DeepSeek and MiniMax document none, so a client needs its own backoff for those.

Why did my SDK not retry a 429?

If it is the Google GenAI or Mistral Python SDK, retries are off by default: pass retry_options or a RetryConfig. If it is the OpenAI Python SDK and the server asked for more than 120 seconds, the SDK deliberately does not retry. If the 429 is a quota or spend cap, not retrying was the right outcome.

Related: billing units field guide, long-context pricing tiers, token usage anatomy, gateway cache audit.

← Back to blog