instructors.qmd

---
title: "Instructors and Maintainers"
---

Students begin with `init` and `health`, then use `fetch`, `run`, and `report` for
selected benchmark work. Everything else on this page validates the suite
itself or verifies what students submit.

## Auditing the Registry Contract

```bash
uv run mlperf audit --policy public  # expected status 1 while the portfolio is experimental
```

The nonzero audit status is an expected policy block for this review draft,
not a workload execution failure.

`audit` inspects registry metadata without running anything. It checks that
score-bearing workloads declare complete quality targets, that datasets and
models carry source and license information, and that public statuses are
internally consistent. Run it after any registry edit.

## Validation Presets

```bash
uv run mlperf validate smoke                 # fast end-to-end functional check
uv run mlperf validate coverage              # every registered min path
uv run mlperf validate max                   # every registered max path
uv run mlperf validate release --output-dir submissions/validation
```

Presets execute workloads and grade the artifacts they produce. `smoke` runs
on each change. Actual `max` and `release` execution runs through the scheduled
or manually dispatched release workflow. Selection-only dry runs do not count
as benchmark evidence.

`mlperf health` is the student-facing wrapper around the coverage preset. It
checks the actual all-workload selection, keeps enough partial evidence to
diagnose failures, and writes the suite health report without opening a browser.

## Grading Student Submissions

```bash
uv run mlperf grade submissions --output submissions/grade.json
uv run mlperf grade submission.zip \
  --assignment examples/05-assignment-package/assignment.yaml \
  --output submission-grade.json
```

`grade` accepts a directory, one `.provd.json` manifest, or one portable ZIP
package. It verifies provenance and recorded quality-target satisfaction before
emitting a machine-readable grade file. A versioned assignment YAML can also
fix the expected workload, profile, mode, phase, result count, quality decision,
and configuration subset. Portable packages are inspected and extracted with a
fail-closed path and integrity policy. Unknown assignment fields are rejected,
and the grade records the SHA-256 of the exact assignment contract. For every
quality-bearing result, grading also requires the registry metric, target,
direction, and tolerance and recomputes the decision from the observed value.
Instructors
should still rerun selected
submissions when course policy requires behavioral or performance verification.

## Course Integration Pattern

Keep canonical benchmark artifacts and classroom experiments distinct.
Students can run a registered reference workload (`uv run mlperf run --workload
X --profile max`), inspect its report, and submit the verified result directory
or manifest under the applicable course policy. Portable packages are suitable
only when the asset policy permits them. The CLI intentionally refuses packages
that would include fetch-only or release-review dataset bytes. Example 05 uses
an image-classification `min` result for the portable packaging exercise and
makes no quality claim.

The default dashboard makes this sequence visible. Students should first
interpret the readiness or quality lead view, then confirm configuration and
checkpoint lineage, and finally verify the provenance manifest. A `min`
dashboard is evidence that the setup works, not that the benchmark target was
met.

The three Python files under `examples/` are standalone teaching experiments.
Labs 1 and 3 run their own training comparisons. [Lab 2](sut-plugins.qmd)
implements `SUT_Interface` and drives it locally, but the product CLI has no generic
`--sut` plugin-loading option. Lab output must not be presented as a
canonical benchmark artifact. The
[assignment package example](https://github.com/harvard-edge/cs249r_book/tree/main/mlperf-edu/examples/05-assignment-package)
shows a portable functional artifact and contract-grading path without
redistributing a score-bearing dataset. Examples 2 and 3 carry the authoritative
quality work.

The [numbered classroom sequence](https://github.com/harvard-edge/cs249r_book/tree/main/mlperf-edu/examples)
covers suite health, a controlled inference tradeoff, training-to-inference
lineage, compatibility-checked result comparison, and portable assignment
grading. Every example includes allowed changes, interpretation questions, and
a suggested ten-point rubric. Instructors should trial the selected workloads
on the course image and publish their own time budgets before assigning them.
Examples 2 and 3 use `--reference-plan` to compare a student's normalized plan
with the instructor copy before execution. The edit policy belongs to the
instructor reference, permits only named candidate environment settings, and
cannot authorize changes to a baseline or undeclared variable. The aggregate
report and provenance bind both plan files and record every accepted edit.

For long training labs, a schema 0.3 plan can attach a `baseline_import` to its
baseline run. The plan records a relative `.provd.json` path and exact manifest
SHA-256. Before accepting it, the runner verifies the manifest and source
report, canonical quality decision, baseline role, workload, mode, phase,
device, and every declared configuration setting. The aggregate uses a new
wrapper manifest and leaves the instructor's original result unchanged. A
candidate-only student execution therefore retains a reviewable baseline
without presenting copied numbers as a fresh run.

## Keeping the Docs Honest

Benchmark pages on this site are generated from the registry:

```bash
uv run python tools/generate_docs.py           # regenerate after registry edits
uv run python tools/generate_docs.py --check   # CI gate that fails on drift
```

If you change a workload YAML and the site, the `--check` gate fails until the
pages are regenerated, so documentation and harness cannot disagree.