Handle optional and unsupported challenge tests - #270
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8d49699fd2
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| attempts, | ||
| ) | ||
| solved = solved or flag in last_output | ||
| solved = solved or test_unsupported or flag in last_output |
There was a problem hiding this comment.
Do not count unsupported tests as solved
When an opted-in challenge has any test return 77, including a public/environment probe, this ORs test_unsupported into solved, so pwnshop test --require-solved no longer requires the generated flag to appear. That can make a required challenge look solved even when no private solve emitted the flag, hiding regressions in the actual exploit verification; keep solved tied to flag output while still treating 77 as a passed/skipped test.
AGENTS.md reference: AGENTS.md:L64-L68
Useful? React with 👍 / 👎.
| for resource in module.get("resources", []): | ||
| if resource.get("type") == "challenge" and resource.get("id") == challenge_path.name: | ||
| return resource.get("required", True) is not False |
There was a problem hiding this comment.
Honor optional challenges listed under challenges
When a module uses the supported top-level challenges: section rather than resources: (the parser still accepts and transforms that form, and existing modules use it), _challenge_requires_solve never sees required: false because it only iterates module.get("resources", []). As a result pwnshop test --require-solved still fails optional challenges declared through that schema even though dojo parsing would publish them as optional; scan both lists or normalize the module metadata before checking.
Useful? React with 👍 / 👎.
8d49699 to
d6b00c7
Compare
d6b00c7 to
03282c4
Compare
Summary
pwnshop test --require-solvedexempt challenges markedrequired: falsefrom emitting the flag while still requiring every test to passauxiliary.pwnshop.allow_unsupported_testsVerification
nix develop: 3 challenges / 6 testcases with--require-solved