infra

infra

The control plane: event topology and tenant provisioning, as Terraform.

Runs against LocalStack by default and against real AWS by flipping one variable. The same configuration serves both — if the LocalStack wiring lived in a separate copy, the thing tested locally would not be the thing deployed, and every drift between them would be found in production.

make up       # LocalStack + terraform apply + write the gateway's config
make verify   # assert the topology is actually correct
make chaos    # prove the redrive policy really dead-letters
make down     # destroy everything, remove state

What it provisions

SNS local-usage    ──► SQS local-usage-metering   ──► DLQ (maxReceiveCount=3)
SNS local-audit    ──► SQS local-audit-audit      ──► DLQ
                   └─► SQS local-acme-audit-export   (enterprise only, filtered)
SNS local-anomaly  ──► SQS local-anomaly-anomaly  ──► DLQ
                       (filter: type ∈ {rate.throttled, bulkhead.shed})

DynamoDB local-tenants        tenant limits — the source of truth
SSM  /tenant-platform/…/api-key   one slot per tenant; values issued out of band

Decisions worth knowing

One queue per consumer, not one per topic. A shared queue makes consumers compete — each message goes to exactly one reader — so a slow audit consumer would eat events metering also needed. Per-consumer queues give each an independent cursor and an independent backlog.

max_receive_count is declared once. It lives in variables.tf and is consumed by every queue. It must agree with the gateway's consumer retry budget: if they drift, an event either dies before SQS gives up (silent loss) or is retried far more than the runbook says. The gateway logs a warning at boot when they disagree, and make verify fails on it.

Filtering happens at the topic, not in the consumer. The anomaly subscription filters on the type message attribute, so the ML router is never billed for — and never has to skip past — ordinary successful requests. Per- tenant audit export filters on tenant_id, which makes the isolation real: another tenant's record is never delivered, so a consumer bug cannot leak it.

Terraform writes the gateway's config. local_file.gateway_endpoints emits ../gateway/config/aws-endpoints.json with the ARNs and queue URLs. Queue URLs are not knowable before apply, and a human copying them into an env file is how local and deployed configuration diverge.

API key values are not in Terraform. The SSM parameter is provisioned with a placeholder and ignore_changes = [value]. Terraform owns the slot and the access path; the credential issuer owns the secret. Without ignore_changes, the next apply would helpfully overwrite a real key.

Connecting the gateway

make -C infra up                     # provisions and writes the handoff file
cd ../gateway
EVENT_BACKEND=sqs make run           # reads config/aws-endpoints.json

Events now flow over real SNS/SQS. The console's DLQ panel and redrive button operate on actual queues; retry and dead-lettering are SQS's job rather than the in-process bus's.

Verification, not vibes

terraform apply proves the API accepted the calls. These prove the topology does what it claims:

Check	Catches
make verify	Missing subscriptions (a topic that accepts publishes and discards them), absent redrive policies, maxReceiveCount drift, empty tenant registry, missing gateway handoff
make chaos	A redrive policy that is present but not effective — wrong DLQ ARN, or a message that never actually moves

Both exit non-zero on failure and are safe to run in CI.

Targeting real AWS

terraform apply -var use_localstack=false -var environment=staging

Before doing that: switch the backend in versions.tf to S3 with DynamoDB locking, and set prevent_destroy = true on aws_dynamodb_table.tenants — losing that table means losing the definition of who may call the platform at what rate, and it is not reconstructible from application logs.

Status

Applied and verified against LocalStack. make demo provisions 30 resources, asserts the topology, and passes the redrive drill.

Verified end to end:

• 3 SNS topics, 4 SQS queues (one per consumer plus Acme's dedicated audit export), 4 DLQs, all with maxReceiveCount=3 matching the gateway's retry budget
• The anomaly filter policy actually filters: with no tenant throttled the anomaly consumer received 0 events while metering and audit each received 179; once a tenant was throttled it received 1,426. The subscription carries {"type":["rate.throttled","bulkhead.shed"]}.
• The redrive policy is effective, not just present: a message received 4× without deletion lands in the DLQ and leaves no duplicate behind
• The gateway boots against these queues with EVENT_BACKEND=sqs and the handoff file, with no code change

Runs on OpenTofu in a container — no host CLI install needed.