Reconcile benchmark readiness checklist

Author: profvjreddi

mlperf-edu/DATASET_RELEASE_REVIEW.md

@@ -67,10 +67,12 @@ and evaluator without project choices.
 
 ## Reviewer Checklist
 
-- [ ] Every active dataset has an authoritative upstream source.
-- [ ] Every revision, file, and split is pinned and verified.
+- [x] Every active dataset has an authoritative upstream source.
+- [x] Every revision, file, and split is pinned and verified, or is generated
+  by the pinned run contract.
 - [ ] License and attribution text is accurate.
-- [ ] Fetch-only and redistribution rules are explicit.
-- [ ] Generated reports disclose data mode and asset hashes.
-- [ ] Package tests reject restricted bytes.
-- [ ] No removed workload dataset remains on the public benchmark site.
+- [x] Fetch-only and redistribution rules are explicit and conservative while
+  external decisions remain open.
+- [x] Generated reports disclose data mode and asset hashes.
+- [x] Package tests reject restricted bytes.
+- [x] No removed workload dataset remains on the public benchmark site.

mlperf-edu/INDEPENDENT_AUDIT.md

@@ -110,12 +110,13 @@ highest-risk findings were valid and are now enforced in code and tests.
   evidence trees or unambiguous source artifacts. All five labs are rendered in
   the website navigation.
 
-The repeated review supports a supervised pilot for Labs 01, 02, 04, and 05.
-Lab 03 is explicitly a take-home accelerator exercise until the product can
-bind a precomputed instructor baseline into a candidate-only plan. Automated
-allowed-plan-diff grading and measured course-image resource budgets also remain
-open. These are product-readiness limits, not reasons to change the fourteen
-benchmark identities or lower their quality targets.
+The repeated review now supports a supervised pilot for all five labs. Lab 03
+can bind a provenance-verified instructor baseline into a candidate-only plan,
+and the CLI enforces allowed plan edits against the instructor reference.
+Measured functional CPU and accelerator-requested resource budgets are
+published for the first course system. Authoritative max budgets on each
+selected course image remain open. This limit is not a reason to change the
+fourteen benchmark identities or lower their quality targets.
 
 ## Primary Sources
 

mlperf-edu/PRODUCT_READINESS_PLAN.md

@@ -150,7 +150,11 @@ research-environment workloads.
 ### Production release
 
 - [ ] Close component license, naming, and governance decisions.
-- [ ] Harden generated-code execution and untrusted pickle handling.
+- [x] Harden generated-code execution and verify trusted upstream pickle bytes
+  immediately before loading them in the controlled preview.
+- [ ] Replace executable EDM pickle inputs, qualify signed legacy runtime
+  images, and close the remaining production controls in
+  [SECURITY_REVIEW.md](SECURITY_REVIEW.md).
 - [ ] Sign release artifacts and provenance where producer identity matters.
 - [ ] Reproduce on independent CPU, Apple Silicon, and CUDA systems.
 - [ ] Run the later stability campaign and promote compatible baselines.

mlperf-edu/QUALITY_TARGET_REVIEW.md

@@ -47,8 +47,8 @@ number means.
 
 This distinction is important for the four current near-misses. PatchTST,
 HumanEval+, BFCL, and EDM remain below strict published-reference points. Their
-results do not justify lowering the gates. The causal-language-modeling point,
-graph tolerance, and causal-language-modeling interpretation remain conditional
+results do not justify lowering the gates. The graph tolerance and the
+causal-language-modeling reproduction-point interpretation remain conditional
 until independently approved. The BFCL data, leaderboard result, and evaluator
 are now pinned to immutable revisions.
 

mlperf-edu/WORK_CHECKLIST.md

@@ -56,20 +56,30 @@ promotion, and production publication are separate later phases.
 These items refine or approve quality interpretation. They do not justify
 lowering a target to fit one local result.
 
-- [ ] Investigate the PatchTST result of 0.292393 MSE against the unchanged
-  target of at most 0.290.
-- [ ] Investigate the HumanEval+ result of 91 passing tasks against the
-  unchanged requirement of at least 94 of 164.
-- [ ] Analyze the BFCL category gaps behind the 0.785208 result against the
-  unchanged 0.8292 target.
-- [ ] Review the EDM sampler and numerical path behind the 1.801554 minimum FID
-  against the unchanged target of at most 1.79.
+- [x] Investigate the PatchTST result. Two exact Apple Silicon reproductions
+  reached 0.29168 and 0.29239 MSE. Neither met the unchanged 0.290 point, and
+  no post-result tolerance was introduced.
+- [x] Investigate the HumanEval+ result. Attention, dtype, Transformers, Qwen
+  evaluator, and EvalPlus variants were cross-checked. The authoritative run
+  still passed 91 of 164 tasks against the unchanged 94-task requirement.
+- [x] Analyze the BFCL category gaps. Java and JavaScript account for the
+  largest deficits, while Python, multiple, and parallel-multiple meet or
+  exceed the corresponding published category scores. The aggregate remains
+  0.785208 against the unchanged 0.8292 target.
+- [x] Review the EDM sampler and numerical path. Three complete 50,000-image
+  trials were independently rehashed and rescored, the official 18-step and
+  35-evaluation schedule was preserved, and the evaluator cross-check differed
+  by only 2.53e-8 FID. The best result remains 1.801554 against 1.79.
 - [ ] Obtain domain approval for the one-sided OGB GCN target interpretation.
 - [ ] Obtain independent approval for the nanoGPT target interpretation.
-- [ ] Resolve the keyword-spotting adapter's quality-preserving but nonidentical
-  promotion boundary.
-- [ ] Close dataset redistribution and fetch-only wording before public
-  release.
+- [x] Resolve the keyword-spotting adapter boundary. Retain it as a disclosed,
+  quality-preserving educational adaptation and block promotion until an
+  exact-source execution path or authoritative unquantized weights establish
+  parity.
+- [x] Use conservative fetch-only wording and reject restricted dataset bytes
+  from portable packages while decisions remain open.
+- [ ] Close the remaining external dataset redistribution and component-terms
+  decisions before public release.
 
 ## Classroom and Research Follow-Up
 
@@ -86,7 +96,9 @@ lowering a target to fit one local result.
   evidence for research interpretation.
 - [ ] Perform an explicitly requested interactive review of a standalone run
   dashboard. Layout automation already passes and no browser should open as a
-  side effect of routine verification.
+  side effect of routine verification. The July 18 attempt stopped without
+  opening anything because the in-app browser runtime reported no available
+  browser.
 
 ## Stability and Promotion
 
@@ -103,8 +115,10 @@ Production publication is not required for the initial classroom and research
 preview.
 
 - [ ] Close component licensing, package naming, versioning, and governance.
-- [ ] Complete security review of generated-code execution, untrusted pickle
-  handling, and legacy container paths.
+- [x] Harden and document the controlled-preview generated-code and pinned
+  pickle paths in [SECURITY_REVIEW.md](SECURITY_REVIEW.md).
+- [ ] Replace executable EDM pickle inputs with safe reviewed artifacts and
+  qualify independently built, signed DLRM and MiniGo runtime images.
 - [ ] Close MLCommons review of the project name, scope, and result wording.
 - [ ] Sign release artifacts and authenticate provenance where producer
   identity matters.