Respond to an anomaly
Triage a spend or usage anomaly and act on it.
AOCore watches spend and usage for statistically unusual movement and raises an anomaly when a metric departs from its recent baseline. Anomalies are informational — they never block traffic. Budgets do that.
Steps
1. Open /anomalies.
Each row names the metric, the observed value, the expected baseline and the deviation that triggered it.
2. Establish whether it is real. A z-score flags departure from a baseline, which is not the same as a problem. Ask, in order:
- Is the baseline meaningful? A metric with only a few days of history has a wide, unstable baseline and produces noise.
- Did something legitimate change — a launch, a migration, a new customer?
- Is it concentrated? Open
/spendand break the window down by model, user, team and tag. A genuine incident is almost always concentrated in one dimension; broad uniform growth is usually adoption.
3. Confirm against the raw rows. /spend-logs holds the individual requests
behind the aggregate. This is where a runaway retry loop becomes obvious — the
same request shape at high frequency from one key.
4. Act.
| Cause | Action |
|---|---|
| Runaway client | Lower the key’s RPM/TPM, or disable the key |
| Cost shifted to an expensive model | Constrain the key’s model allowlist |
| Legitimate growth | Raise the budget; acknowledge the anomaly |
| Compromised key | Rotate it — see Issue and scope an API key |
5. Acknowledge so it leaves the active list and the record shows who assessed it.
Getting told without watching
/alert-rules defines conditions that raise alerts; /alert-events shows what
fired and its acknowledgement state. Alerts dispatch by email, Slack, webhook or
in-app.
Set a soft budget below the hard ceiling as the earliest useful signal — it alerts without refusing traffic, which on an organization matters because hitting the hard limit stops every team beneath it simultaneously.
Anomalies you can safely ignore
A quiet key going to zero. Absence of traffic is flagged as readily as excess.
The first days after onboarding. Nothing has a baseline yet.
Spikes that match a deploy. Correlate the timestamp with /audit-logs before
investigating further — a configuration change is a likelier explanation than an
attack.
When the numbers themselves look wrong
Dashboard and analytics figures come from a pre-aggregated daily rollup for complete calendar days, with today overlaid live from the raw logs. So:
- Complete days reconcile exactly with
/spend-logs. A discrepancy there is a real defect worth reporting. - Today will not, and should not — it is a live partial sum that moves as traffic arrives.
Comparing today’s rollup figure against a raw query and finding them different is expected behaviour, not a bug.