Make your model API calls survive rate limits, timeouts, and partial failures

October 14, 2026 · 7 min read

SentX SI

Work disappears from model API integrations for one reason: the client treats every failure the same. A transient rate-limit error, a transient overload, a quota error that never clears on its own, and a stream that died halfway through are all "an exception" to code that catches and moves on — or that crashes and loses the item entirely. The fix is a decision method rather than more retries: classify each failure before acting, and make sure every request ends in a recorded state — succeeded, deferred, failed with a record, or partially completed with a record. Nothing is silently dropped when nothing is allowed to vanish without a trace. This guide follows OpenAI's documented rate-limit behavior; the shape of the method transfers, but other providers define their own codes and headers, and their documentation is where you get those values.

Tell the two transient errors apart

OpenAI's documentation distinguishes two transient conditions that look alike in a log but demand different recovery. Rapid traffic increases return HTTP 429 with the error code slow_down; temporary model overload returns HTTP 503 with server_is_overloaded. The client must branch on status plus error code, not status alone, because other errors reuse both statuses.

Source: Rate limits | OpenAI API

One property of slow_down surprises people: it can fire even when you are inside your requests-per-minute and tokens-per-minute limits. It measures how fast your traffic rose, not whether you exhausted the limit — so the correct response is to slow the ramp, not just to wait.

Source: Rate limits | OpenAI API

In the official SDKs, these map to distinct exception types, so handlers can branch on exception as well as status: Python, TypeScript, and Ruby raise RateLimitError for 429 and InternalServerError for 503; Java raises RateLimitException and InternalServerException.

Source: Rate limits | OpenAI API

Treat Retry-After as a floor, then bound the fallback

When a valid Retry-After header is present, wait at least that long and add a small random delay so concurrent clients do not re-collide on the same second. When the header is missing or invalid, fall back to exponential backoff with jitter — start short, double after each failed attempt, add randomness — and cap both the number of attempts and the total wall-clock time spent retrying. Two further rules from the documentation keep this from misbehaving in production:

Source: Rate limits | OpenAI API

1. Official SDKs already retry eligible 429 and 503 responses. Check the retry behavior of your installed version, and if you manage retries in your application, disable SDK retries or fold them into your limits so nested loops do not multiply requests.Source: Rate limits | OpenAI API
2. If a valid server-supplied delay exceeds your configured maximum retry delay, stop retrying and defer the request rather than retrying sooner.Source: Rate limits | OpenAI API

A compact fallback loop, adapted from the manual example in the documentation:

Source: Rate limits | OpenAI API

import random
import time
from openai import OpenAI, RateLimitError, InternalServerError

client = OpenAI()

def with_backoff(func, initial_delay=1.0, base=2.0, max_retries=6,
                 errors=(RateLimitError, InternalServerError)):
    delay = initial_delay
    for attempt in range(max_retries):
        try:
            return func()
        except errors:
            if attempt == max_retries - 1:
                raise
            delay = delay * base * (1 + random.random())
            time.sleep(delay)

The documentation's own examples deliberately stop at this fallback path: they do not read Retry-After, so a production wrapper adds that check in front of the backoff and prefers the server's number whenever it is valid.

Source: Rate limits | OpenAI API

Never retry errors that need you

Quota, billing, and other user-action errors are not transient. The documentation is explicit that Retry-After does not mean such errors can be resolved by waiting, and it tells clients not to retry them. A hard spend limit, for instance, surfaces as a 429 that no amount of backing off will clear. These errors should propagate out of the retry layer immediately, land in a place an operator will see, and stop the affected workload — retrying them only burns budget against a wall that will not move.

Source: Rate limits | OpenAI API

Streaming fails in a different shape

For streaming requests, the HTTP error response applies before the stream starts. Once the stream is flowing, an error can arrive as a stream event, mid-output. The rule the documentation gives is blunt: do not automatically replay a request after consuming output. A correct handler keeps whatever it received, records the item as partially completed, and decides what to do with the remainder — reissue only the unfinished piece, or flag it for review — instead of firing the whole request again and doubling what already happened.

Source: Rate limits | OpenAI API

Timeouts are a separate axis

A per-attempt timeout is not a deadline for the whole operation, and a canceled request or expired deadline can stop retries without ever returning the HTTP error your handler is watching for. Handle cancellation and timeout explicitly as their own outcomes, with their own recorded states, or they become another silent hole.

Source: Rate limits | OpenAI API

Why tight loops make it worse

Failed requests still consume your per-minute quota, so continuously resending a rejected request cannot work — every retry competes with your real traffic for the same budget. Two levers reduce pressure at the source: set max_tokens close to the response size you actually expect, since the limit calculation takes the larger of that value and the estimated input tokens, and move work that does not need an immediate answer to the Batch API, which runs large collections without drawing on your synchronous rate limits.

Source: Rate limits | OpenAI API

On ramping new traffic, the documentation offers a rule of thumb, flagged as approximate: once you reach 1 million input tokens per minute, grow by no more than 50% every 15 minutes, because the exact point at which ramp-rate limiting kicks in varies by model and conditions.

Source: Rate limits | OpenAI API

When the endpoint isn't OpenAI's

The method travels; the constants don't. Each provider defines its own statuses, error codes and headers, so a client built for one has to re-read the documentation of every other before its handler can trust a single number. Worth checking early is whether the provider publishes its current limits at all. SentX SI, the Dubai-based research and product company behind the Victoria model, does: its public model card states that availability, daily limits and media allowances depend on the current service and plan, and it points integrators to the current policies and API documentation before wiring anything up. That is the same habit this guide keeps repeating — the published page is the source of truth, and it will change under you.

Source: Victoria AI Model Card | Capabilities, Evidence and Limits | SentX SI

A failed request, traced end to end

This walkthrough is an illustration built from the documented behavior, not an observed result. A nightly enrichment job sends 300 prompts:

1. The job launches at full concurrency. Within seconds, several calls return 429 / slow_down with Retry-After: 56. The client waits at least 56 seconds plus a random offset, cuts concurrency, and ramps back up gradually.Source: Rate limits | OpenAI API
2. An hour in, a call returns 503 / server_is_overloaded with no usable Retry-After. The client falls back to capped exponential backoff with jitter; the attempt budget exhausts before the item succeeds, so the item is written to a dead-letter list with its error — visible, retryable tomorrow, not gone.Source: Rate limits | OpenAI API
3. A different call fails with a billing error. It bypasses the retry layer entirely and halts the job with a message an operator sees.Source: Rate limits | OpenAI API
4. A streaming call delivers part of its output, then a stream event reports the failure. The client keeps the partial output, marks the item partially completed, and reissues only the unfinished portion.Source: Rate limits | OpenAI API

Every one of the 300 items ends in a state someone or something can find. That invariant — no request may disappear without a record — is what separates a resilient client from one that quietly drops work, and it costs less than the first silent data loss would.

Provider-specific numbers in OpenAI's rate-limit documentation change over time, so check the current page before you encode any of them as constants into code.

Source: Rate limits | OpenAI API