Hardcap is an LLM gateway that enforces per-tenant token budgets across any number of replicas without overshooting. The load tests that prove it ship with the code.
Admits while used < limit, charges when each stream ends. Every in-flight stream is invisible to the budget.
Reserves before the call, grows the reservation while the stream runs, and ends a stream cleanly when the budget is spent.
An LLM request can cost 10 tokens or 100,000, and output length is only known when the stream ends. So most gateways check used < limit on admission and add the real count afterwards. Between those two moments, a request is invisible to the budget: with 128 streams in flight, all 128 get admitted while the tenant still looks under budget. More replicas make it worse.
Each step is a single Lua script in Redis. Redis runs one script at a time, so for any tenant the checks and updates are serialized, however many gateway replicas send them.
Checks the request rate, reclaims reservations left by crashed replicas, and reserves input plus estimated output tokens. One round trip; a rejected request costs nothing.
-- atomic, per tenant if used + reserved + est > limit then return busy | exceeded reserved += est
The estimate is the rolling p95 of real output lengths. As a stream nears its reservation, the gateway asks for 64 more tokens. If the budget can't cover them, the stream ends with finish_reason: "length".
-- every 64 tokens if used + reserved + 64 > limit then truncate reserved += 64
The reservation is swapped for the usage the provider reported, and cost is attributed per tenant, route and model in Prometheus.
-- when the response ends reserved -= reservation used += actual
Checked inside every admit and extend. Settlement swaps a reservation for the real count, so the budget can only be exceeded by tokens generated without a reservation, and truncate mode allows none.
8 tenants × 3 runs per point, a 20,000-token budget each, log-normal output lengths. Each tenant's clients stream requests until the gateway says the budget is spent.
| At 128 streams per tenant | Worst overshoot | Budget usable | Truncated / tenant |
|---|---|---|---|
| Count after the fact (typical gateway) | +234.3% | 100% | 0 |
| Reserve max_tokens | never over | 95.6% | 0 |
| Reserve rolling p95, allow overrun | +7.9% | 98.7% | 0 |
| Reserve rolling p95, truncate (Hardcap default) | +0.03% | 98.7% | 1.2 |
The +0.03% (5 tokens) is real: it's what the provider generates between the gateway closing a stream and the provider noticing. Reserving max_tokens never overshoots, but it holds the full cap for every in-flight stream, so about 4% of each budget can never be used.
| Load | Added p50 | Added p99 |
|---|---|---|
| 1,000 req/s | +1.1 ms | +1.2 ms |
| 2,500 req/s | +1.6 ms | +3.2 ms |
| 1,000 req/s, streaming | +1.9 ms | +3.0 ms |
Five Python workers saturate between 2,500 and 5,000 req/s on a 12-thread laptop shared with the load generator, mock upstream and Redis.
| Primary upstream | Failed |
|---|---|
| Returns 503 for 7 s | 0 / 4,000 |
| Killed with SIGKILL | 26 / 4,000 |
All 26 were streams already sending bytes from the killed process, which no gateway can retry. Every request sent after the kill succeeded.
Raw results, machine specs and every parameter are in the repo. Rerun any benchmark with one command.
Input, output and total limits per window, enforced across stateless replicas.
Sliding-window counter or token bucket, checked atomically with the budget.
budget_busy (retry in 1 s) is kept separate from budget_exceeded (retry when the window resets).
Ordered fallback on 429, 5xx, timeouts and connection errors, with a breaker per upstream.
An OpenAI-compatible API in front of OpenAI, vLLM and other compatible servers, plus Anthropic, translated both ways.
Prometheus metrics per tenant, route and model, plus a ready-made Grafana dashboard.
The built-in mock LLM lets you try budgets, truncation and fallback locally. Then point any OpenAI SDK at Hardcap and use a tenant key as the API key.
# install and run against the mock LLM pip install hardcap docker run -d -p 6379:6379 redis:7-alpine curl -sO https://raw.githubusercontent.com/saikirankalidindi/hardcap/main/examples/local.yaml hardcap mock --port 9001 & hardcap serve -c local.yaml
# any OpenAI client works from openai import OpenAI client = OpenAI(base_url="http://localhost:8080/v1", api_key="sk-alpha") # a tenant key client.chat.completions.create( model="demo", stream=True, messages=[{"role": "user", "content": "hi"}])
# config.yaml: one tenant, one route with fallback budget: {accounting: reserve, estimator: p95, on_overrun: truncate} upstreams: openai: {kind: openai, base_url: https://api.openai.com, api_key_env: OPENAI_API_KEY} anthropic: {kind: anthropic, base_url: https://api.anthropic.com, api_key_env: ANTHROPIC_API_KEY} routes: default: {targets: [{upstream: openai, model: gpt-4o-mini}, {upstream: anthropic, model: claude-haiku-4-5}]} tenants: search-team: {api_keys: [sk-hc-search], budget: {window_s: 3600, total: 2000000}}