fix(corpus): key the total-failure guard on failures, not on skips - #471
Open
JOhnsonKC201 wants to merge 1 commit into
Open
fix(corpus): key the total-failure guard on failures, not on skips#471JOhnsonKC201 wants to merge 1 commit into
JOhnsonKC201 wants to merge 1 commit into
Conversation
The guard added in calesthio#466 exempts any run that contains a skipped clip: total_failure = bool(candidates_seen) and not added_ids and not skipped `skipped` counts clips that were already in the corpus, so it says nothing about whether the candidates we actually processed succeeded. A run that skips one already-present clip and then fails the other nine on a broken decoder or CLIP stack returns success with clips_added: 0 and error: None, which is the same silent-empty-index failure the guard was added to prevent. The comment already states the intended rule, "a no-result or skip-only run remains valid", and a skip-only run is one where failed == 0, so key the condition on failed instead of on skipped. Also make the diagnostic accurate when prior clips are present, since "corpus index is empty" is false in that case. Adds two regression tests: total failure alongside skips must fail closed, and a skip-only run must stay successful.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #466. That PR added a fail-closed guard so a corpus build that processes candidates and embeds none of them stops reporting success. The guard is right, but it is keyed on
skipped, which disables it as soon as a single already-present clip shows up in the run.skippedcounts clips that were already in the corpus. It says nothing about whether the candidates we actually processed succeeded. So a run that skips one already-present clip and then fails the other nine on a broken decoder or CLIP stack returnssuccess=True,clips_added: 0,error: None, which is exactly the silent-empty-index failure the guard was added to prevent.The existing comment already describes the intended rule: "a no-result or skip-only run remains valid." A skip-only run is one where
failed == 0. This keys the condition on that instead.Related issue
Refs #466, #424
Changes
tools/video/corpus_builder.py: key the guard onfailed > 0rather thannot skipped, so already-present clips no longer suppress a total processing failure.tools/video/corpus_builder.py: make the diagnostic accurate when prior clips exist."corpus index is empty"is false in that case, so the message now reportsno clips added (N already present). The original wording is unchanged whenskipped == 0.tests/tools/test_corpus_builder_total_failure.py: extend therun_builderfixture with an optionalexistingargument that marks clip ids as already present, and add two regression tests.Behavior is unchanged for every case #466 already covered.
Testing
Verified the new test fails on current
mainbefore the fix, with the exact silent success described above:After the fix:
Both tests added by #466 pass unchanged. Nothing else in the tree asserts on the old error wording.
Platform: Windows, Python 3.13.
Checklist
make test-contracts/make test) where applicable.