Artificial Intelligence / API reliability / Implementation guide
Handle AI API rate limits by checking which budget you exceeded, pacing new requests through a shared queue, and retrying only recoverable failures with a deadline and attempt limit. Requests per minute, tokens per minute, daily quotas and spending controls are different constraints; increasing concurrency does not increase those allowances.
A prototype may work with one person and fail when a scheduled import starts beside live customer traffic. The useful response is to measure the combined workload and control admission, rather than asking every failed request to try again immediately.
Record the provider, endpoint, model, account or project, current limits and observed error details. This guide uses Claude and Gemini documentation as examples. The calculations and queue policy below are hypothetical planning tools, not measured throughput or universal provider settings.
01 / Separate rate, size and spending budgets
A rate limit describes activity over time. A context or request-size limit describes a single call. A spending control describes an account’s allowed expenditure. Fixing one does not automatically fix the others.
| Constraint | Question to answer | Planning response |
|---|---|---|
| Request rate | How many calls may this shared budget admit? | Pace starts, including retries. |
| Token rate | Which input or output tokens count over the window? | Estimate token demand before admission. |
| Daily or longer quota | Is the total allowance exhausted? | Defer work until access returns or review the plan. |
| Single-request size | Does this call fit the model and endpoint? | Change the request rather than repeatedly resending it. |
| Spending control | Is more use permitted under the account’s budget? | Escalate for a deliberate account decision. |
For the separate problem of fitting a long document into one call, see our context window vs output limit guide. Here the focus is admitting many calls without overwhelming a shared allowance.
02 / Read the limits for your actual account
Gemini API documentation describes requests per minute, input tokens per minute and requests per day, with additional model-specific limits. These apply per project rather than per API key. Its active limits are available in AI Studio, and stated limits are not a guarantee of available capacity.
For Claude’s Messages API, Anthropic documents RPM, input tokens per minute and output tokens per minute by model class. Organization-wide limits remain relevant even when workspaces have their own settings. It also explains that short bursts can trigger limits despite an acceptable minute-level average.
Build a small configuration record containing the current values and where they came from. Add the exact counting rules, reset or replenishment behaviour, model group and any workspace restriction. Recheck this record when changing models or account tiers.
Do not copy a limit table from an old tutorial into a permanent scheduler. A sample account’s allowance is not evidence of your entitlement. Likewise, creating another key within the same quota scope should not be treated as new capacity.
Keep token accounting provider-specific. For example, caching and output accounting rules can differ by model. Use observed usage and the current documentation to validate estimates instead of assuming every provider combines all tokens in the same way.
03 / Estimate the first bottleneck
Consider a hypothetical service with these invented limits and workload assumptions:
- 60 requests per minute.
- 120,000 input tokens per minute.
- 12,000 output tokens per minute.
- Each job uses about 4,000 input tokens and 600 output tokens.
Under a simplified steady-load calculation, the request allowance permits 60 jobs per minute, input permits 30, and output permits 20. Output is the first modeled bottleneck, so adding enough workers to start 60 jobs each minute would not solve the capacity problem.
Request ceiling: 60 / 1 = 60 jobs/minute
Input ceiling: 120000 / 4000 = 30 jobs/minute
Output ceiling: 12000 / 600 = 20 jobs/minute
Simplified ceiling = minimum(60, 30, 20)
= 20 jobs/minuteThis calculation assumes stable job sizes and separate input/output accounting. It excludes bursts, competing applications, retries, enforcement details and service capacity. It is a starting point for an admission policy, not a target to run against without margin.
If ten of those twenty jobs instead produce 1,200 output tokens, the original estimate is too optimistic. Measure the distribution of job sizes and reserve headroom for larger requests. Avoid basing the whole system on one unusually short sample response.
Also count the calls inside a job. A workflow that classifies, retrieves, drafts and reviews may issue several model calls for one user action. Put those calls in the capacity estimate separately, including any repair pass.
04 / Pace request starts and cap work in flight
Use an admission queue that represents the shared budget. A single process may hold it locally; multiple workers need coordination at the scope where they compete. Decide how to keep one large import from consuming all capacity needed by interactive users.
Rate and concurrency controls solve different scheduling problems. Pacing controls when work starts. A concurrency cap controls how many requests are active. Keep both: slow responses can accumulate even when new starts are carefully spaced.
As a hypothetical example, admitting one request every three seconds corresponds to twenty starts per minute in a simple evenly spaced schedule. If each call takes twelve seconds, roughly four may overlap in steady conditions. That arithmetic describes the example, not the concurrency you should configure for every API.
Give queue entries an expiry time and a stable job identifier. If a user cancels or an interactive deadline passes, stop admitting obsolete work. For background processing, show a durable queued state and let an operator inspect failures without restarting the whole import.
If the task can wait, evaluate the provider’s asynchronous batch service on its own terms. Our batch API vs real-time guide covers that design choice. Moving to a batch service still requires checking its separate quotas and job handling.
05 / Retry only within a bounded policy
Google’s Gemini troubleshooting guidance recommends exponential backoff, jitter and maximum retry counts for transient errors. It also notes that official SDKs can already retry automatically. Inspect your installed SDK’s behaviour before adding an application retry loop around it.
Anthropic’s rate-limit documentation says to wait the specified number of seconds when a response supplies retry-after. Respect that signal and the job’s remaining deadline; do not keep an interactive request waiting indefinitely just because another retry is technically possible.
The following is illustrative pseudocode. It assumes the provider error has already been classified and any retry delay parsed correctly. It is not executable SDK code:
until attempt budget is exhausted:
if job deadline has passed:
stop with an explicit timeout result
wait for admission through the shared scheduler
send one request
if successful:
persist result and finish
if failure is not recoverable:
stop and report the reason
choose provider-required delay when available
otherwise choose bounded backoff with jitter
if delay would exceed the job deadline:
stop or defer the background job
schedule the next attemptKeep retry ownership explicit. If both the SDK and the application retry, the number of network attempts can exceed the number visible in the outer loop. Budget actual attempts, and ensure the admission control sees them.
Amazon’s retry-design discussion explains why retries can add load to an already struggling service and why a timeout does not prove a remote operation had no side effects. For AI workflows, persist generation results before later actions and protect database writes or tool calls from duplicate execution.
06 / Diagnose the response before buying more capacity
Claude’s current error reference distinguishes invalid requests, authentication, permissions, billing, rate-limit and overloaded-service errors. It also documents a spend-cap case that returns 429 without a retry-after header. Therefore, the HTTP number alone is not enough to choose the recovery action.
| Observed situation | Investigation | Application action to consider |
|---|---|---|
| Temporary throttling with a retry signal | Identify the exhausted budget and delay. | Pause through the scheduler within the deadline. |
| Daily allowance or spend cap reached | Check account status and reset or approval requirements. | Defer work and notify the responsible operator. |
| Bad key, permissions or malformed request | Inspect configuration and request validation. | Stop unchanged retries and correct the cause. |
| Timeout or temporary service failure | Check request ID, service status and partial work. | Apply the documented recovery policy and duplicate protection. |
Log the provider request ID when available, model, job ID, attempt count, status, queue delay and token usage. Keep secrets and unnecessary document content out of diagnostic logs. Use these records to answer whether failures coincide with larger prompts, a scheduled job, another application or a specific model.
Request a quota increase only after identifying the constrained budget and the expected workload. Higher allowances do not repair an invalid request or an uncontrolled retry policy.
07 / Verify behaviour before launch
Use a small approved workload and controlled failure simulation to check the scheduler. This is a suggested verification plan; no load test or benchmark was performed for this article.
- Confirm all workers using a shared allowance participate in admission control.
- Check ordinary jobs and larger jobs, including input and output estimates.
- Simulate a retry delay and verify the application does not retry early.
- Simulate a nonrecoverable account error and verify attempts stop.
- Test job cancellation, queue expiry and application restart.
- Count actual SDK attempts and confirm duplicate side effects are prevented.
Review completion time and queue age as well as error counts. A system can hide throttling by building a backlog that no longer meets the business deadline. Define when to reject new work, postpone an import or give users a useful queued response.
After changing a model or prompt, revisit the estimates. A workflow that produces longer answers may shift the bottleneck even if the number of users stays constant.
Frequently asked questions
Will adding more workers fix rate-limit errors?
It may improve use of available capacity, but it does not raise the shared allowance. Coordinate admission and measure the constrained budget first.
Does every 429 mean I should retry after a few seconds?
No. Inspect the provider error details and account state. A longer quota or spending restriction may require deferral or an account decision.
Can I use one retry loop for every provider?
You can share a scheduling design, but preserve provider-specific error classification, delay signals, token accounting and SDK behaviour.
What should a user see when capacity is unavailable?
Show a clear queued, deferred or failed state that matches the application’s deadline. Preserve accepted work when appropriate and avoid encouraging repeated submissions.
Sources and editorial notes
Primary documentation was checked on 28 October 2026. Limits vary by provider, model, endpoint and account. All numerical workloads and pseudocode are illustrative; they are not quota quotations, production code or measured performance claims.
Add comment