Rate limits & quotas

Every /v1/* response — success and 429 — carries up to six X-AOSentry-* response headers describing the caller’s remaining quota. These header names are part of the published API contract — the same contract the API reference is generated from — and a parity test fails the build if the implementation and the contract ever disagree, in either direction.

The six headers

Absent vs zero — important distinction

absent header and header: 0 mean very different things:

• Absent means “no limit of that type is configured on this key.” Send as much as you want — there’s no gate.
• 0 means “the limit is configured, and you’ve hit it.” The next request will return 429.

So an OpenAI-SDK adapter that wants to react to quota should check if header is present and value <= threshold — not just if value <= threshold (which would trigger on absent keys).

Sample response — quota nearly exhausted

HTTP/2 200
Content-Type: application/json
X-AOSentry-RateLimit-Remaining: 3
X-AOSentry-RPM-Remaining: 3
X-AOSentry-TPM-Remaining: 12450
X-AOSentry-Budget-Remaining: 0.1234
X-AOSentry-RateLimit-Reset: 1715763600
X-AOSentry-Budget-Reset: 2026-06-01T00:00:00Z

This caller has 3 requests left this minute, ~12k tokens left this minute, $0.1234 of budget left, and the per-minute window resets at unix timestamp 1715763600 (≈ in 45 seconds). The budget resets at the start of next month.

429 retry policy

When you hit a limit the gateway returns 429 Too Many Requests:

{
  "error": {
    "message": "Rate limit exceeded — RPM",
    "type": "rate_limit_error",
    "code": "rate_limit_exceeded"
  }
}

The 429 response also carries the six headers — including X-AOSentry-RateLimit-Reset — so you can compute the exact pause:

import time
pause_until = int(response.headers["X-AOSentry-RateLimit-Reset"])
time.sleep(max(0, pause_until - int(time.time())))

The correct client policy is:

1. On 429, look at X-AOSentry-RateLimit-Reset.
2. Sleep until that unix timestamp.
3. Retry.

Do not retry on a fixed backoff — you’ll either retry too soon (wasting an attempt) or too late (idle for longer than needed). The Reset header is the authoritative timestamp.

Budget exhaustion is reported as 429 with code: "budget_exceeded" and is not recoverable by waiting — it requires either a budget increase from the administrator or a fresh budget window (if budget_reset_at is set). Your client should distinguish:

if response.status_code == 429:
    err_code = response.json()["error"]["code"]
    if err_code == "budget_exceeded":
        raise BudgetExceeded(...)  # Don't retry; escalate.
    elif err_code == "rate_limit_exceeded":
        reset = int(response.headers["X-AOSentry-RateLimit-Reset"])
        time.sleep(reset - time.time())
        # retry

Reading the headers from Python (OpenAI SDK)

The OpenAI Python SDK hides the raw response by default. Use with_raw_response to access it:

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://gateway.core.aocyber.ai/v1",
    api_key=os.environ["AOSENTRY_KEY"],
)

raw = client.chat.completions.with_raw_response.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello"}],
)

# AOCore-specific quota headers:
for header in (
    "X-AOSentry-RateLimit-Remaining",
    "X-AOSentry-RPM-Remaining",
    "X-AOSentry-TPM-Remaining",
    "X-AOSentry-Budget-Remaining",
    "X-AOSentry-RateLimit-Reset",
    "X-AOSentry-Budget-Reset",
):
    if header in raw.headers:
        print(f"{header}: {raw.headers[header]}")

# The parsed completion is still available:
completion = raw.parse()
print(completion.choices[0].message.content)

See examples/python-openai-sdk/chat.py for a fully runnable version.

Reading the headers from LangChain

LangChain wraps the OpenAI client, but you can attach a callback that exposes the raw response:

from langchain_openai import ChatOpenAI
from langchain_core.callbacks import BaseCallbackHandler

class QuotaPrinter(BaseCallbackHandler):
    def on_llm_end(self, response, **kwargs):
        # ChatOpenAI sets generation_info from the underlying response on each
        # generation. For the OpenAI SDK, response.llm_output may carry raw
        # headers when configured via http_client — see the example for details.
        ...

# For full header visibility, drop down to the OpenAI SDK directly.

The cleaner approach is to use the OpenAI SDK directly when you need quota headers, and use LangChain for orchestration where the quota headers aren’t needed per-call. See examples/python-langchain/README.md for the recommended pattern.

Configuring limits

You don’t set limits from the client — they’re attached to your minted API key at mint time. In the portal’s key-mint flow you can pick:

• max_budget — dollar cap. The gateway enforces it on every request before upstream call. Once exceeded, all requests on the key return 429 budget_exceeded.
• rpm_limit — requests/minute cap. Enforced via spend_logs count for the last 60 seconds.
• tpm_limit — tokens/minute cap. Enforced via spend_logs.total_tokens sum for the last 60 seconds.
• budget_reset_at — optional RFC3339 reset window (omit for non-resetting budgets).

All three are also clamp-enforced at mint time against your developer’s own limits — you cannot mint a key with a max_budget higher than your developer’s remaining headroom.

Related

• the authentication guide — how to send the key
• the errors guide — full APIError shape including rate_limit_error + budget_exceeded
• the models guide — model allowlists (a different gate type)
• the Quickstart — picking a max_budget at mint time
• API reference — components.headers for the contract source