fix(shipping): don't panic if flagd is unreachable at startup - #3875
fix(shipping): don't panic if flagd is unreachable at startup#3875mwimpelberg28 wants to merge 2 commits into
Conversation
Falls back to open_feature::provider::NoOpProvider when FlagdProvider fails to connect within its retries, matching checkout's non-fatal handling of the same startup race. Fixes open-telemetry#3845. Assisted-by: Claude Sonnet 5
|
I'd rather have the container in crashloop till it can connect with flagd than having the service running, but not working as expected because of NoOp flagd |
|
@julianocosta89 this seems to differ to other services behaviour as per #3845 |
Kielek
left a comment
There was a problem hiding this comment.
I'd rather have the container in crashloop till it can connect with flagd than having the service running, but not working as expected because of NoOp flagd
Fully agree with this, we should wait until flagd is healthy on the docker startup level.
|
@julianocosta89 @Kielek do you want PR to change the checkout service to match this intended behaviour then, is that approach you want to go with? |
|
@dpacheconr the main problem that I see here is the If they were |
Fixes #3845.
shippingpanics and crash-loops if it can't reachflagdwithin its5 startup connection retries (~31s). On Kubernetes there's no ordering
guarantee between the
shippingandflagddeployments, so this iscommon on a fresh install or rollout.
This falls back to
open_feature::provider::NoOpProvider(logging theerror) when
FlagdProvider::newfails, instead of panicking. Thismatches how
checkout(Go) handles the same race — it only logs andkeeps running.
shipping_service.rsalready treats flag-resolutionerrors as non-fatal at the call site (defaults
intlShippingSlowdownto 0), so the service tolerates a missing flag provider everywhere
except this startup path.
Note this fallback is a one-shot decision: if it triggers, that
shippingprocess runs on the NoOp provider for its process lifetime(no background reconnect) until the pod restarts for some other reason.
Tested locally: scaled
flagdto 0 replicas, restartedshipping—it now starts cleanly and serves traffic instead of crash-looping.