Is your feature request related to a problem?
The policy file instructors write today only covers resource access: what files, network connections, subprocesses, threads, and packages a test is allowed to touch. A separate set of behavioral features (deadline timing, output capture, exception handling, test visibility, and how much detail a failure reports back) can currently only be set through per-test annotations scattered across the test code, not through the policy file itself. That split means an instructor has to look in two different places to understand how a given test actually behaves, and none of these behavioral settings are visible to, or overridable from, the one file meant to describe an exercise's security and execution configuration. Concretely, none of the following currently have any representation in the policy format: enforcing a time limit, mirroring a test's console output, pre-configuring how a test's input/output is handled, scheduling when a hidden test activates or extends its deadline, restricting which exceptions are treated as privileged, marking a test as hidden or public, and choosing how much detail a sandbox violation reports back to the student. One related piece already exists in the policy format but doesn't do anything yet: a field for execution-time limits is parsed and validated today, but the underlying execution path it would need to enforce against isn't wired up, so setting it currently has no effect. Worth fixing as part of bringing the equivalent per-test annotation into the policy properly, rather than leaving two silently disconnected copies of the same concept.
Describe the solution you'd like
Add a policy field for each of the following behavioral features, matching what its corresponding per-test annotation already controls today, and make the Ares 2 runtime apply the setting when the policy loads:
- Time-limit enforcement (today's
@StrictTimeout), and connect it to the existing but currently inert time-limit field already present in the policy format so that one mechanism actually does what it's meant to.
- Output mirroring (today's
@MirrorOutput).
- I/O handling pre-configuration and selection (today's
@WithIOManager).
- Hidden-test deadline scheduling, the base deadline (today's
@Deadline).
- Hidden-test extended-deadline scheduling (today's
@ExtendedDeadline).
- The setting that controls when a hidden test activates before its deadline (today's
@ActivateHiddenBefore).
- Restricting which exceptions are treated as privileged (today's
@PrivilegedExceptionsOnly).
- Marking a test as hidden (today's
@Hidden).
- Marking a test as public (today's
@Public).
- The level of detail a sandbox violation reports back to the student.
For every field above, a per-test annotation present on a specific test should still override the corresponding policy-level setting for that test. The policy sets the default; the annotation narrows or changes it for one case. Add validation and clear error reporting for the new fields, consistent with how the existing resource-access fields are validated today, and provide example policies demonstrating the new fields so instructors have a working reference rather than having to reverse-engineer the format from the schema alone.
Describe alternatives you’ve considered
Leaving these as per-test-only annotations. Rejected: it's the specific problem this request exists to close, instructors currently can't see or set a test's full behavior from one file, and consolidating it there is the whole point.
Additional context
The items above naturally split into two independently deliverable batches, should that help sequence the work: time-limit enforcement, output mirroring, I/O handling, privileged exceptions, and feedback level in one batch; the deadline-related fields plus hidden/public visibility in the other. Nothing on this list has any implementation yet; the policy schema (the ResourceAccesses record under policy/policySubComponents/) currently exposes exactly six resource-access categories and none of the behavioral ones.
Is your feature request related to a problem?
The policy file instructors write today only covers resource access: what files, network connections, subprocesses, threads, and packages a test is allowed to touch. A separate set of behavioral features (deadline timing, output capture, exception handling, test visibility, and how much detail a failure reports back) can currently only be set through per-test annotations scattered across the test code, not through the policy file itself. That split means an instructor has to look in two different places to understand how a given test actually behaves, and none of these behavioral settings are visible to, or overridable from, the one file meant to describe an exercise's security and execution configuration. Concretely, none of the following currently have any representation in the policy format: enforcing a time limit, mirroring a test's console output, pre-configuring how a test's input/output is handled, scheduling when a hidden test activates or extends its deadline, restricting which exceptions are treated as privileged, marking a test as hidden or public, and choosing how much detail a sandbox violation reports back to the student. One related piece already exists in the policy format but doesn't do anything yet: a field for execution-time limits is parsed and validated today, but the underlying execution path it would need to enforce against isn't wired up, so setting it currently has no effect. Worth fixing as part of bringing the equivalent per-test annotation into the policy properly, rather than leaving two silently disconnected copies of the same concept.
Describe the solution you'd like
Add a policy field for each of the following behavioral features, matching what its corresponding per-test annotation already controls today, and make the Ares 2 runtime apply the setting when the policy loads:
@StrictTimeout), and connect it to the existing but currently inert time-limit field already present in the policy format so that one mechanism actually does what it's meant to.@MirrorOutput).@WithIOManager).@Deadline).@ExtendedDeadline).@ActivateHiddenBefore).@PrivilegedExceptionsOnly).@Hidden).@Public).For every field above, a per-test annotation present on a specific test should still override the corresponding policy-level setting for that test. The policy sets the default; the annotation narrows or changes it for one case. Add validation and clear error reporting for the new fields, consistent with how the existing resource-access fields are validated today, and provide example policies demonstrating the new fields so instructors have a working reference rather than having to reverse-engineer the format from the schema alone.
Describe alternatives you’ve considered
Leaving these as per-test-only annotations. Rejected: it's the specific problem this request exists to close, instructors currently can't see or set a test's full behavior from one file, and consolidating it there is the whole point.
Additional context
The items above naturally split into two independently deliverable batches, should that help sequence the work: time-limit enforcement, output mirroring, I/O handling, privileged exceptions, and feedback level in one batch; the deadline-related fields plus hidden/public visibility in the other. Nothing on this list has any implementation yet; the policy schema (the
ResourceAccessesrecord underpolicy/policySubComponents/) currently exposes exactly six resource-access categories and none of the behavioral ones.