A multi-tenant inference gateway, and the three services around it.
One customer's traffic must never take down another customer's.
That is the whole setup. Docker is the only prerequisite.
You sell a product with an AI feature. Customers pay you monthly. Behind the scenes, every time a customer uses it, you call OpenAI or Anthropic and you pay that bill.
It's Tuesday. One customer starts a bulk import — 10,000 support tickets to summarise, all at once.
Without something in the middle:
| 1 | Their 10,000 requests consume every connection your service has |
| 2 | Every other customer's AI feature stops working |
| 3 | You get paged |
| 4 | You get a surprise four-figure inference bill |
| 5 | You can't tell which customer caused it, because nothing was metered per-tenant |
This repo is the thing in the middle. Every component maps to one of those five failures.
Who actually runs software like this
This is a real product category: OpenRouter, Portkey, LiteLLM, Cloudflare AI Gateway, Helicone, Kong AI Gateway. And every SaaS that added AI features — Notion, Intercom, Zendesk, Linear — built an internal version, usually after an incident taught them why.
The pattern is older than AI, too. Swap "model providers" for "payment processors" and it's Stripe. Swap for "SMS carriers" and it's Twilio. A multi-tenant API gateway with per-customer limits is one of the most common backend architectures there is.
| In this repo | In the real world |
|---|---|
| Bulkhead | Stops the bulk-import customer eating all your connections |
| Rate limiter | "Free: 5 req/s. Enterprise: 50 req/s." — your pricing page, enforced |
| Circuit breaker | OpenAI has an outage (they do) → fail over to another provider instead of going down with them |
| Quota + metering | The usage-based line on the invoice |
| Tenant console | The customer-facing usage dashboard, like OpenAI's billing page |
| Operator console | Your internal on-call dashboard during an incident |
| Router | Don't spend frontier-model money fixing a typo. A real, large line item. |
Prerequisite: Docker. That's the whole list — Go, Node, Python and OpenTofu all run inside containers.
git clone <this repo> && cd tenant-platform
make upOne command starts four components in dependency order, and tells you what each one is for as it goes. First run pulls images and takes about three minutes; after that it's under a minute.
| Step | What happens, and why it has to be in this order |
|---|---|
| 1 · infra | LocalStack starts, OpenTofu provisions 3 SNS topics, 6 SQS queues, DLQs, and the tenant registry. First, because it writes the queue URLs everything downstream reads. |
| 2 · gateway | The Go data plane and the operator console, wired to the real SNS/SQS spine that step 1 created. |
| 3 · console | The React tenant UI plus its BFF, which seeds each tenant's policy from the gateway — so the gateway has to be up already. |
| 4 · router | The Python routing service, which starts consuming the gateway's live event queues to learn how hard each tenant's traffic is. |
Then:
make seed # push differentiated traffic through all three tenants
make status # what is running, and where
make down # stop everything, remove volumes| URL | What it is |
|---|---|
| localhost:9090/_console start here |
Operator console — your internal dashboard. The buttons induce real failures: flood one tenant, kill an upstream provider, break a consumer and watch its dead-letter queue fill. Each button states what to expect before you press it. |
| localhost:5174 | Tenant console — what a customer sees. Sign in as different users; each sees only their own tenant, and only what their role permits. Open devtools and diff two sessions. |
| localhost:8090/docs | Router API — POST /route returns a decision with its full reasoning attached. |
make up, then open the operator console- Press
initech · 120/s — noisy neighbour - Watch initech's bars fill with purple (
shed) and amber (throttled) - Watch acme's and globex's green bars not change at all. That's the entire thesis
- Press
kill primary— the circuit opens, traffic keeps flowing via the secondary provider - Press
break audit consumer→ the DLQ fills →redrive DLQ→ it drains
graph TB
client([customer])
nginx["nginx edge<br/>per-IP limits · tenant-blind"]
subgraph gw["gateway · Go"]
dp["data plane :8080<br/>quota → bulkhead → rate limit → upstream"]
cp["control plane :9090<br/>operator console"]
end
valkey[("Valkey<br/>shared token buckets")]
providers["3 model providers<br/>a circuit breaker each"]
subgraph spine["event spine · SNS → SQS"]
usage[["usage.events"]]
audit[["audit.events"]]
anomaly[["anomaly.events"]]
end
subgraph consoleapp["console · React + BFF"]
web["web :5174"]
bff["BFF :8788<br/>the isolation boundary"]
end
routersvc["router · Python :8090<br/>cost/quality routing"]
infra["infra · OpenTofu<br/>provisions the spine"]
client --> nginx --> dp
dp --> valkey
dp --> providers
dp --> usage
dp --> audit
dp --> anomaly
usage --> routersvc
anomaly --> routersvc
routersvc -. "learned difficulty" .-> dp
web --> bff --> cp
infra -.->|"tofu apply"| spine
infra -.->|"writes queue URLs"| gw
classDef comp fill:#4A7BD6,stroke:#2c5aa0,color:#fff
class gw,consoleapp,routersvc,infra comp
| Component | Stack | What it owns |
|---|---|---|
gateway/ | Go | Data plane. Per-tenant rate limiting, bulkheads, circuit breakers, failover, metering. Ships the operator console. |
infra/ | OpenTofu + LocalStack | Control plane. SNS→SQS topology, redrive and filter policies, tenant provisioning. |
console/ | React + TS + Node BFF | Tenant UI. Server-enforced RBAC, server-side downsampling, conflict-aware policy editor. |
router/ | Python | Learned routing. Contextual bandit, off-policy evaluation, drift monitor, staged rollout. |
Most gateways ship a rate limiter and call multi-tenancy solved. It isn't:
A rate limiter bounds arrivals. A bulkhead bounds residency. A slow upstream attacks residency, and the rate limiter never fires.
When a provider degrades from 200 ms to 20 s, a customer sending a constant, perfectly legal 5 req/s goes from 1 concurrent request to 100. Their rate never changed — so the limiter sees nothing wrong while they consume every connection in the process and everyone else times out.
flowchart TD
req([request]) --> auth{"1 · authenticate"}
auth -->|"unknown key"| r401["401"]
auth --> quota{"2 · quota<br/>atomic read"}
quota -->|"over budget"| r402["402 · quota"]
quota --> bulk{"3 · bulkhead<br/>bounds IN-FLIGHT"}
bulk -->|"too many concurrent"| r503["503 · shed"]
bulk --> rate{"4 · rate limit<br/>bounds ARRIVALS"}
rate -->|"bucket empty"| r429["429 · throttled"]
rate --> up{"5 · upstream<br/>breakers + failover"}
up -->|"all providers down"| r502["502 · upstream"]
up --> ok["200 · streamed"]
classDef bad fill:#C4433F,stroke:#8f2f2c,color:#fff
classDef key fill:#7C5FC8,stroke:#553f8c,color:#fff
classDef good fill:#2F8A54,stroke:#1f5d39,color:#fff
class r401,r402,r502 bad
class r503,r429 key
class ok good
Stage 3 runs before stage 4 deliberately: rate-limiting first means a
customer whose requests all hang still burns a slot per arrival. Pinned by
TestBulkheadContainsSlowUpstreamThatRateLimiterCannot, which asserts requests
were shed and that zero were throttled — if the limiter ever starts
catching this case the test fails, because it is no longer testing what it claims.
One queue per consumer, never one per topic. SQS delivers each message to exactly one reader, so a shared queue makes consumers compete and each silently sees half the traffic.
graph LR
gw["gateway"]
gw --> T1[["SNS usage.events"]]
gw --> T2[["SNS audit.events"]]
gw --> T3[["SNS anomaly.events"]]
T1 --> Q1["SQS metering"]
T1 --> Q2["SQS router"]
T2 --> Q3["SQS audit"]
T2 -->|"filter: tenant_id"| Q4["SQS acme-audit-export"]
T3 -->|"filter: type"| Q5["SQS anomaly"]
T3 -->|"filter: type"| Q6["SQS router"]
Q1 --> C1["metering consumer<br/>quota state"]
Q3 --> C3["audit consumer"]
Q5 --> C5["anomaly consumer"]
Q2 --> R1["router · difficulty"]
Q6 --> R2["router · pressure"]
Q1 --> D1{{"DLQ"}}
Q3 --> D3{{"DLQ"}}
Q5 --> D5{{"DLQ"}}
classDef dlq fill:#B5842C,stroke:#7d5c1e,color:#fff
class D1,D3,D5 dlq
Filtering happens at the topic, not in the consumer. The per-tenant audit
export filters on tenant_id, which makes the isolation real: another tenant's
record is never delivered, so a consumer bug cannot leak it.
Every queue has a DLQ with maxReceiveCount = 3, declared once in
infra/variables.tf and matched against the gateway's retry budget — the
gateway warns at boot if they disagree, and make -C infra verify fails on it.
The gateway's :9090 holds every tenant's numbers plus the chaos endpoints. It
must never be reachable from a customer's browser.
sequenceDiagram
participant B as Browser
participant W as nginx :5174
participant F as BFF :8788
participant G as gateway :9090
B->>W: GET /api/overview + Bearer token
W->>F: proxy
F->>F: resolve token to Principal
F->>F: require usage:read
F->>G: GET /api/state
G-->>F: ALL tenants, providers, DLQ
Note over F: projectState builds a NEW object:<br/>drop other tenants<br/>drop cost unless cost:read<br/>drop traces unless trace:read<br/>never send providers / bus / DLQ
F-->>B: one tenant, no internals
The BFF never sends a byte the principal is not entitled to, and the browser never filters anything.
Projection constructs rather than deletes: a field the gateway starts
returning next year is hidden by default, not exposed by default. A test adds an
internal_margin_usd field upstream and asserts it never reaches the client.
Cross-tenant access returns 404, not 403 — a 403 confirms the tenant exists and turns the endpoint into a customer-enumeration oracle.
graph LR
A["gateway serves<br/>a request"] --> B[["usage.events<br/>tokens"]]
A --> C[["anomaly.events<br/>throttled / shed"]]
B --> D["difficulty<br/>EWMA per tenant"]
C --> E["pressure<br/>EWMA per tenant"]
D --> F["context vector"]
E -.->|"surfaced,<br/>not yet acted on"| F
F --> G["LinUCB picks the cheapest<br/>model that is good enough"]
G --> H["decision + propensity<br/>logged"]
H --> I["off-policy evaluation<br/>of the NEXT policy"]
I -.-> G
classDef live fill:#2F8A54,stroke:#1f5d39,color:#fff
class D,E live
Measured live, end to end, through real SNS→SQS:
| tenant | traffic | learned difficulty | pressure |
|---|---|---|---|
| acme | 250-token responses | 0.362 | 0.0 |
| globex | 20-token responses | 0.054 | 0.0 |
| initech | flooded at 120 rps | 0.119 | 0.837 |
Two signals because they imply different actions: hard traffic argues for a more capable model; a customer already being throttled argues for a faster one, since adding latency to a backlog is how a slowdown becomes an outage.
The design constraint that has to be settled before any data exists: the router logs the probability with which it made each choice. It cannot be reconstructed afterwards, and without it you can never evaluate a replacement policy from logs — only with a live A/B test on real customers.
stateDiagram-v2
[*] --> shadow
shadow --> canary1: 2000 samples clean
canary1 --> canary5: clean
canary5 --> canary25: clean
canary25 --> full: clean
canary1 --> rolled_back: 200 samples breach
canary5 --> rolled_back: 200 samples breach
canary25 --> rolled_back: 200 samples breach
rolled_back --> [*]: needs a human
note right of shadow
decide and log,
but do not act
end note
note right of rolled_back
terminal on purpose:
a flapping policy must not
oscillate unattended
end note
The asymmetry is the point. Promoting late costs some savings; promoting a bad policy costs quality on live customer traffic, and every extra sample spent confirming it is a customer getting a worse answer.
Nothing below is estimated. Every row was run.
| Claim | Verify with |
|---|---|
| Token bucket: 159 ns/op, 0 allocs (contended) | make -C gateway bench |
| A tenant at 40× its limit causes zero throttling for neighbours | make -C gateway test |
| A slow upstream is contained by the bulkhead, not the limiter | make -C gateway test |
| Dead upstream → failover, circuit opens, recovery unattended | make -C gateway chaos |
| Hand-rolled RESP client works against real Valkey 8 | make -C gateway up |
| Redrive policy is effective, not merely present | make -C infra chaos |
| SNS filter policy actually filters | make -C infra demo |
| A session's payload contains no trace of another tenant | make -C console test |
| A latency spike survives downsampling | make -C console test |
| Bundle within budget — 66.9 kB gz of 120 kB | make -C console bench |
| Three OPE estimators land within 0.002 of ground truth | make -C router bench |
| Cost/quality frontier: 55% cheaper at −0.027 quality | make -C router bench |
Isolation tests are mutation-checked in both the gateway and the console: deliberately breaking the isolation makes them fail. A test that passes is only worth something if it can fail.
Stated plainly, because a portfolio that blurs this is worth less than one that doesn't.
|
Real code, production-shaped
|
Simulated
|
That line is deliberate. You cannot put a real provider key in a public repo, and you cannot demo an upstream outage on request — so the thing being proxied is fake while the proxy is real. Every component README says so in its own Status section.
Every component implements the same targets. That consistency is a platform property, not a cosmetic one — it is what lets one entrypoint orchestrate four languages.
| Target | Guarantee |
|---|---|
make up |
Running and healthy in under 90 s, seeded, from a clean checkout |
make demo |
Scripted walkthrough. Prints URLs. Requires no prior reading |
make test |
Full suite, race detector where the language has one |
make bench |
Reproduces every number claimed in that component's README |
make chaos |
Induces failure and asserts recovery — exits non-zero on failure |
make down |
Full teardown, no orphans |
Each records the options rejected and why. The rejections carry more information than the decisions.
- ADR-0001 — Quota is enforced asynchronously
- ADR-0002 — Two-tier rate limiting
- ADR-0003 — Valkey over Redis
- ADR-0004 — Take the AWS SDK, having refused a Valkey driver
- ADR-0005 — OpenTofu, run from a container
ADR-0003 and ADR-0004 reach opposite conclusions about dependencies on purpose. The rule that reconciles them:
Hand-roll a protocol only when its full surface is small, frozen, and fails loudly. Import it when correctness depends on details that fail silently or only in production.
RESP passes that test. SigV4 does not.
gateway/README.md— the constraint, and arrivals vs. residencymake up, then watch localhost:9090/_console while pressing the noisy-neighbour buttongateway/internal/gateway/gateway.go— the request path; the check ordering is the design- ADR-0002 — why three limiting tiers, and what each one cannot do
console/README.md— why the BFF, not the browser, is the isolation boundaryrouter/README.md— why a router must log its own propensity
| Port | Service |
|---|---|
| 8000 | nginx edge → data plane |
| 8080 | gateway data plane |
| 9090 | operator console — never publicly routable |
| 5174 | tenant console |
| 8788 | console BFF |
| 8090 | router — OpenAPI docs at /docs |
| 4566 | LocalStack |