Skip to content

fix(shipping): don't panic if flagd is unreachable at startup - #3875

Open
mwimpelberg28 wants to merge 2 commits into
open-telemetry:mainfrom
mwimpelberg28:fix-shipping-flagd-panic
Open

fix(shipping): don't panic if flagd is unreachable at startup#3875
mwimpelberg28 wants to merge 2 commits into
open-telemetry:mainfrom
mwimpelberg28:fix-shipping-flagd-panic

Conversation

@mwimpelberg28

Copy link
Copy Markdown
Contributor

Fixes #3845.

shipping panics and crash-loops if it can't reach flagd within its
5 startup connection retries (~31s). On Kubernetes there's no ordering
guarantee between the shipping and flagd deployments, so this is
common on a fresh install or rollout.

This falls back to open_feature::provider::NoOpProvider (logging the
error) when FlagdProvider::new fails, instead of panicking. This
matches how checkout (Go) handles the same race — it only logs and
keeps running. shipping_service.rs already treats flag-resolution
errors as non-fatal at the call site (defaults intlShippingSlowdown
to 0), so the service tolerates a missing flag provider everywhere
except this startup path.

Note this fallback is a one-shot decision: if it triggers, that
shipping process runs on the NoOp provider for its process lifetime
(no background reconnect) until the pod restarts for some other reason.

Tested locally: scaled flagd to 0 replicas, restarted shipping
it now starts cleanly and serves traffic instead of crash-looping.

Falls back to open_feature::provider::NoOpProvider when FlagdProvider
fails to connect within its retries, matching checkout's non-fatal
handling of the same startup race. Fixes open-telemetry#3845.

Assisted-by: Claude Sonnet 5
@julianocosta89

Copy link
Copy Markdown
Member

I'd rather have the container in crashloop till it can connect with flagd than having the service running, but not working as expected because of NoOp flagd

@dpacheconr

Copy link
Copy Markdown
Contributor

@julianocosta89 this seems to differ to other services behaviour as per #3845

@Kielek Kielek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather have the container in crashloop till it can connect with flagd than having the service running, but not working as expected because of NoOp flagd

Fully agree with this, we should wait until flagd is healthy on the docker startup level.

@dpacheconr

Copy link
Copy Markdown
Contributor

@julianocosta89 @Kielek do you want PR to change the checkout service to match this intended behaviour then, is that approach you want to go with?

@julianocosta89

Copy link
Copy Markdown
Member

@dpacheconr the main problem that I see here is the NoOp part.
AFAIK the other services start and keep trying to connect till it does.

If they were NoOp the flags would be flaky.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

shipping panics and crash-loops if flagd isn't reachable within 5 startup retries

4 participants