fix(capabilities): thread experiment_id through is_stale - #1280
Open
stateofkate wants to merge 1 commit into
Open
fix(capabilities): thread experiment_id through is_stale#1280stateofkate wants to merge 1 commit into
stateofkate wants to merge 1 commit into
Conversation
_load_fresh_analysis passes experiment_id to is_stale, which does not accept it, so every cache lookup that finds a stored block raises TypeError. The durable analyzer job burns all six attempts on it and the next poll enqueues a fresh one, so a task version that already carries a comparison can never regenerate -- the pane polls forever and renders it as an indefinite 'Analyzing agent behavior...' spinner. Seen in prod on scarf-cargotracker-quarkus-to-spring-migration v2 (four consecutive FAILED jobs, six attempts each) and 05-F3-poison-message- recurrence v23. The cache-lookup tests all stubbed the row out as None, which returns before the freshness comparison, so nothing covered the hit path.
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Oddish previewCommit:
Vercel deployment URL: https://oddish-98w069qpg.oddish.app Plan:
This comment is updated by the PR Preview workflow. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
_load_fresh_analysisforwardsexperiment_idtois_stale, which has no suchparameter:
The call is guarded by
row is not None, so it only fires on a cache hit —which is why nothing caught it.
load_stored_analysisusesstale_reasondirectly and is unaffected, so the read paths look fine; the damage is confined
to
get_or_generate_analysis, i.e. the durable analyzer job.Consequence: a task version that already carries a stored comparison can never
regenerate. The job dies on
TypeErrorfor all six attempts, and becauseenqueue_analysisonly treats QUEUED/RUNNING/RETRYING/BLOCKED as active, thenext 3-second poll enqueues a fresh job — which fails the same way. The pane
has no error state for this, so it renders
data === nullas an indefinite"Analyzing agent behavior across successful and failing runs…" spinner while
burning six attempts per cycle.
Observed in prod (
worker_jobs, modeagent_capabilities):scarf-cargotracker-quarkus-to-spring-migrationv2attempts=6,TypeError: is_stale() got an unexpected keyword argument 'experiment_id'scarf-cargotracker-jakarta-to-spring-migrationv205-F3-poison-message-recurrencev23The fix
Give
is_staletheexperiment_idparameter and forward it tostale_reason,which already understands it. Scope becomes part of freshness for the boolean
helper too, matching what
stale_reasonhas done since #1240.Tests
Three new tests exercise
_load_fresh_analysisagainst a session that returns astored block, the half every existing cache-lookup test stubbed out as
None:All three fail on
stagingwith the TypeError above and pass here.The one failure,
test_summary_warmup_regenerates_truthy_stale_xai_summary, ispre-existing on
staging(verified on a clean checkout of 005002b) anduntouched by this change.