Skip to content

Handle optional and unsupported challenge tests - #270

Merged
zardus merged 1 commit into
mainfrom
pwnshop-optional-unsupported-solves
Aug 1, 2026
Merged

Handle optional and unsupported challenge tests#270
zardus merged 1 commit into
mainfrom
pwnshop-optional-unsupported-solves

Conversation

@zardus

@zardus zardus commented Aug 1, 2026

Copy link
Copy Markdown
Member

Summary

  • make pwnshop test --require-solved exempt challenges marked required: false from emitting the flag while still requiring every test to pass
  • treat exit status 77 as unsupported only for challenges explicitly opting in through auxiliary.pwnshop.allow_unsupported_tests
  • avoid retrying an accepted unsupported result and teach the local dojo parser to validate and preserve arbitrary challenge auxiliary data

Verification

  • targeted formatter check: clean
  • 8 focused unit tests, including malformed metadata and a parser round-trip
  • all 7 active dojo manifests parse successfully on the integration stack
  • affected speculative-execution levels passed under nix develop: 3 challenges / 6 testcases with --require-solved
  • complete integration matrices previously passed twice: Orange 183/366, Yellow 162/324, Green 95/201

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8d49699fd2

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

attempts,
)
solved = solved or flag in last_output
solved = solved or test_unsupported or flag in last_output

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Do not count unsupported tests as solved

When an opted-in challenge has any test return 77, including a public/environment probe, this ORs test_unsupported into solved, so pwnshop test --require-solved no longer requires the generated flag to appear. That can make a required challenge look solved even when no private solve emitted the flag, hiding regressions in the actual exploit verification; keep solved tied to flag output while still treating 77 as a passed/skipped test.

AGENTS.md reference: AGENTS.md:L64-L68

Useful? React with 👍 / 👎.

Comment on lines +43 to +45
for resource in module.get("resources", []):
if resource.get("type") == "challenge" and resource.get("id") == challenge_path.name:
return resource.get("required", True) is not False

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Honor optional challenges listed under challenges

When a module uses the supported top-level challenges: section rather than resources: (the parser still accepts and transforms that form, and existing modules use it), _challenge_requires_solve never sees required: false because it only iterates module.get("resources", []). As a result pwnshop test --require-solved still fails optional challenges declared through that schema even though dojo parsing would publish them as optional; scan both lists or normalize the module metadata before checking.

Useful? React with 👍 / 👎.

@zardus
zardus force-pushed the pwnshop-optional-unsupported-solves branch from 8d49699 to d6b00c7 Compare August 1, 2026 06:04
@zardus
zardus force-pushed the pwnshop-optional-unsupported-solves branch from d6b00c7 to 03282c4 Compare August 1, 2026 06:18
@zardus
zardus merged commit 783c3a8 into main Aug 1, 2026
5 checks passed
@zardus
zardus deleted the pwnshop-optional-unsupported-solves branch August 1, 2026 06:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant