Evaluator: add native per-task repeats/attempts for Fabric Agent Eval targets

[puhuim]

Problem

Agent Eval now exposes per-task/per-metric trial values with stable trial_id identity, and Harbor/Gym targets can natively request repeated executions:

• Harbor: n_attempts
• Gym: num_repeats

The Fabric target has no equivalent run-level repeat/attempt field. A Fabric evaluation therefore produces one trial per task. Consumers that need repeated measurements currently have to submit multiple Eval Jobs or duplicate tasks externally.

This is especially visible for Fabric-based skill evaluation: the same Agent Eval dataset and metric can be reused, but variance, flaky-task analysis, pass@k, and baseline/candidate comparisons require external orchestration even though the result model already supports multiple trials per task.

Current behavior

FabricRunnerTarget supports:

• config
• model
• timeout_s
• capture_trajectory
• skills

It does not expose an attempts/repeats setting.

Submitting one Fabric Eval Job with N tasks produces one trial per task. To obtain R attempts, a coordinator must create R jobs (and then aggregate across jobs) or manufacture duplicate tasks, which is undesirable because task identity is intended to represent the test case rather than an execution attempt.

Proposed behavior

Add a native run-level field to the Fabric target, preferably using one canonical name across agent runtimes, for example:

{
  "kind": "fabric",
  "n_attempts": 3,
  "config": {},
  "skills": []
}

For each task, the Fabric runtime should execute the agent n_attempts times and emit one independently identified trial per attempt.

Requirements

• Preserve one stable task/test-case identity across attempts.
• Generate a unique trial_id for each attempt.
• Preserve attempt index and Fabric/Relay provenance in trial metadata.
• Capture output and ATIF/ATOF trajectory independently for every attempt.
• Score every attempt independently.
• Populate summary.task_metric_values with every trial.
• Make the resulting trials usable by existing task-outcome and pass@k analysis.
• Define retry semantics separately from statistical attempts: a transport/runtime retry must not become an additional scored attempt.
• Respect max_concurrent_tasks and avoid multiplying concurrency unexpectedly.
• Persist partial results when some attempts fail.
• Document resume/idempotency behavior for interrupted jobs.

API and compatibility

• Default to one attempt so existing Fabric jobs are unchanged.
• Add the field to the Agent Eval OpenAPI schema and SDK models.
• If possible, converge the Harbor, Gym, and Fabric naming in a follow-up without breaking existing fields.

Acceptance criteria

1. A Fabric job with 2 tasks and 3 attempts produces 6 trials.
2. Each task has three distinct trial_id values in summary.task_metric_values.
3. Failed attempts are represented as missing/failed trial values without dropping successful sibling attempts.
4. ATIF trajectory evidence is retained per attempt.
5. Existing aggregation and pass@k helpers operate without an external cross-job merge.
6. Coverage includes API validation, runtime execution, persistence, and result-summary tests.

Integration context

TestCrew currently exposes agents through API targets and also validates Fabric-based skill injection using Agent Eval. Native Fabric attempts would remove coordinator-owned repeat scheduling while retaining Platform as the owner of trial identity, scoring, aggregation, and lineage.