Separate vaults for hidden tests
Description:
To prevent advanced models from cheating by reading previous commits, we need to completely isolate the holdout tests. Either encrypt the hidden tests within the git tree or store them in a separate remote vault, fetching them only for the human to edit or for the sandbox test runner to execute.
Acceptance Criteria:
- Agent container has zero file-system or git access to the actual test assertions.
- Test runner successfully fetches and executes the isolated tests.
Separate vaults for hidden tests
Description:
To prevent advanced models from cheating by reading previous commits, we need to completely isolate the holdout tests. Either encrypt the hidden tests within the git tree or store them in a separate remote vault, fetching them only for the human to edit or for the sandbox test runner to execute.
Acceptance Criteria: