Fix CI flakiness: restart gateway on wedged reload, retry deployment once - #415
Open
ilijabojanovic wants to merge 1 commit into
Open
Fix CI flakiness: restart gateway on wedged reload, retry deployment once#415ilijabojanovic wants to merge 1 commit into
ilijabojanovic wants to merge 1 commit into
Conversation
Traced repeated CI failures (federation, plugin-python-grpc, analytics-splunk, ...) to the same root cause: a transient network error during a gateway reload's Dashboard policy fetch can wedge the gateway's reload processor entirely, since it only processes one reload at a time. Every reload queued behind it - including the periodic resends added in #412 - gets accepted but never actually runs, so wait_for_api_loaded's retry budget always runs out. wait_for_api_loaded now restarts the tyk-gateway container once if resending reloads doesn't clear things after 40s, then gives it one more fresh retry budget before failing for real. Also wrap deployment creation in the test workflow with one down.sh + up.sh retry, to absorb transient bootstrap failures unrelated to the code under test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
federation,plugin-python-grpc,analytics-splunk(and occasionally others) to a single root cause: a transient network error during a gateway reload's Dashboard policy fetch can wedge the gateway's reload processor entirely, since it only processes one reload at a time. Every reload queued behind it — including the periodic resends added in Fix/wait for api loaded retry budget #412 — gets accepted but never actually runs, sowait_for_api_loaded's retry budget always runs out.wait_for_api_loaded(scripts/common.sh) now restarts thetyk-gatewaycontainer once if resending reloads hasn't cleared things after ~40s, then gives it one more fresh retry budget before failing for real.tyk-demo-tests.ymlnow wrapsup.shwith onedown.sh+up.shretry, to absorb transient bootstrap failures unrelated to the code under test.Investigation details
Confirmed via gateway container logs from a failed run:
"Network error detected, attempting to re-register node..." context="policy fetch"fires mid-reload, and noreload: completeis ever logged again for the rest of the job. Subsequent group-reload requests (including the #412 resends) log"Group reload accepted"but never actually execute, because they're queued behind the wedged one.Test plan
bash -n scripts/common.shpasses (done locally)Tyk Demo Testsworkflow viaworkflow_dispatchon this branch and confirmfederation/plugin-python-grpc/analytics-splunkpass🤖 Generated with Claude Code