Receive: writes for existing tenants block for the entire WAL replay since v0.41.0 (#8562) instead of failing fast

[mattshep]

Thanos version: v0.42.4 (regression introduced in v0.41.0 by #8562; still present on main)
Object storage: S3
Deployment: receive in router/ingestor split mode, RF=2, ketama, 10 ingestors (~4M head series each), --receive-forward-timeout=15s, --receive.forward.async-workers=50

What happened

When a receive ingestor restarts and its pod is recreated quickly (same node, PVC-pinned), its gRPC listener accepts connections ~10s after start while its TSDBs replay WAL for 2.5–3 minutes. During that whole window, every router→ingestor forward for a replaying tenant is accepted and then blocks server-side until the router's --receive-forward-timeout expires. The routers' per-peer forward worker pools (workers + equal-sized queue) fill within seconds and each request holding series for that peer then parks a goroutine in the blocking sendWrite path for up to the timeout, retaining the decoded write request. At our scale this degrades the routers enough to produce 30–116 rps of client-facing 503s for ~2.5 minutes per restarting ingestor (measured on a 10-ingestor prod roll: 7% of requests 5xx over 47 minutes).

Two details make it worse than a plain slow peer:

1. The blocked forwards fail with DeadlineExceeded, and the router's peer backoff only engages on codes.Unavailable (pkg/receive/handler.go:1029-1033 at v0.42.4), so markPeerUnavailable never fires and routers keep dispatching to the replaying peer at full rate for the entire replay.
2. The asymmetry is easy to observe: when the pod lands on a new node (old IP dead), dials fail fast with Unavailable, backoff engages, and we measure zero client errors for the same class of restart.

What we expected

A write for a tenant whose TSDB is not yet ready should fail fast (as it did through v0.40.1), with a retryable status, so the sender's quorum/retry machinery and the router's peer backoff can do their jobs.

Root cause

#8562 ("receive/multitsdb: always block", first released in v0.41.0) changed MultiTSDB.TenantAppendable from the non-blocking getOrLoadTenant(tenantID, false) (v0.40.1 pkg/receive/multitsdb.go:819-820, with the blockingStart gate at :808) to always joining the tenant-init singleflight:

// v0.42.4 pkg/receive/multitsdb.go:381-388
func (m *MultiTSDB) initTSDBIfNeeded(tenantID string, t *tenant) error {
	_, err, _ := m.initSingleFlight.Do(tenantID, func() (any, error) {
		...
		err := m.startTSDB(logger, tenantID, t)   // -> tsdb.Open() == WAL replay

On cold start, MultiTSDB.Open() loads every on-disk tenant through this same singleflight, and the gRPC server starts serving concurrently with it (cmd/thanos/receive.go:312-313: "Start all components while we wait for TSDB to open"). So an incoming RemoteWrite for a pre-existing tenant mid-WAL-replay joins the in-flight startTSDB call and blocks for the full replay — singleflight.Do is not context-aware, so nothing bounds it server-side.

#8562's motivation (#8446, "First write always fails in Receive") was real, but the errors quoted in its description are telling:

got status code: 500 ... rpc error: code = Internal desc = rpc failed: get appender: TSDB not ready

Internal there is itself a bug: receive's own ErrNotReady (multitsdb.go:1110-1111) is a different instance from Prometheus's tsdb.ErrNotReady, so the fast-path check in pkg/receive/writer.go:84 and the isNotReady mapping (handler.go:1212-1216) never match it, and the not-ready error surfaced as non-retryable Internal instead of Unavailable. Had it been Unavailable, the #8446 first-write failure would have been retried gracefully by the sender and absorbed by quorum — i.e. the proper fix for #8446 was the error code, not making the write path blocking.

Proposed fix

• TenantAppendable blocks only when the tenant has no on-disk data (the genuine First write always fails in Receive #8446 case — creating an empty TSDB is fast); for an existing tenant whose startTSDB is in flight (WAL replay, minutes), return ErrNotReady immediately.
• Map receive's ErrNotReady to codes.Unavailable (fix the sentinel mismatch in writer.go:84 / isNotReady), so senders retry and the router-side peer backoff (markPeerUnavailable) engages.

Happy to send a PR restoring these semantics if the direction sounds right.

Workaround we're deploying meanwhile

Router-side gRPC client health checking via --receive.grpc-service-config (#7907): the receive gRPC server already registers grpc.health.v1 on service "" tied to the readiness prober (NOT_SERVING for the replay window), so

{"loadBalancingConfig":[{"round_robin":{}}],"healthCheckConfig":{"serviceName":""},"methodConfig":[{"name":[{}],"retryPolicy":{"maxAttempts":3,"initialBackoff":"0.1s","maxBackoff":"1s","backoffMultiplier":2,"retryableStatusCodes":["UNAVAILABLE"]}}]}

CAUTION for anyone copying this: retryPolicy.maxBackoff is REQUIRED — grpc-go rejects the entire service config as invalid without it, and because the config is applied via WithDefaultServiceConfig, grpc.NewClient for every forward connection then fails, taking down ALL forwarding (we verified this the hard way in our staging environment; fail-fast validation of this flag at startup might be worth considering upstream). A minimal config without methodConfig also validates and provides the fail-fast behavior alone. When valid, this makes forwards to a replaying ingestor fail locally with Unavailable (which also engages the peer backoff) instead of being absorbed for the forward timeout — same shape as the query path's EndpointGroupGRPCOpts default (#8559). Caveat: the statusProber.Healthy() → health.Resume() race at listener-open (#8693) leaves a few seconds where health reads SERVING before the hashring event flips it to NOT_SERVING.

Related

#8446 (motivation for #8562), #8710 / #8720 (router-side symptom of the same restarts, fixed in 0.42.0 — bounds the damage but each forward still burns a worker slot for the full timeout because the server absorbs instead of rejecting), #8776 (read-path readiness during startup), #8693 (gRPC health SERVING race at startup).