Skip to content

eval_csv_troubleshooting_examples.md

Latest commit

 

History

History
163 lines (110 loc) · 4.8 KB

eval_csv_troubleshooting_examples.md

File metadata and controls

163 lines (110 loc) · 4.8 KB
Raw

Eval CSV Troubleshooting Examples

Use this page when python -B scripts/dev.py eval-csv output is missing, stale, surprising, or about to be used as release evidence. Read it with docs/demo_report.md, docs/command_output_troubleshooting_map.md, docs/eval_authoring_guide.md, and docs/local_artifact_glossary.md.

The core rule: eval_summaries.csv is generated eval evidence, not source truth. Keep it separate from source content unless a release process explicitly asks for a reviewed evidence artifact.

Expected CSV Output

Run:

python -B scripts/dev.py eval-csv

Expected success signal:

Wrote 3 row(s) to <repo-root>\eval_summaries.csv

Expected columns:

  • project
  • total_cases
  • passed_cases
  • pass_rate
  • unsafe_failures

Expected source of truth:

  • eval fixtures: checked-in project data/eval_cases.json
  • eval summaries: generated by each project scripts/run_eval.py
  • CSV output: generated eval_summaries.csv
  • public metrics narrative: docs/demo_report.md and README evidence sections

Missing CSV Output

Symptom:

  • eval_summaries.csv is missing after a release review.
  • A contributor expects CSV output but only ran python -B scripts/dev.py evals.
  • The command exits before writing rows.

Wrong fix:

  • Create eval_summaries.csv by hand.
  • Commit a copied CSV from another branch.

Safe fix:

python -B scripts/dev.py eval-csv
git status --short --branch

If the command fails, inspect the project eval runner named in the error before changing source data. Do not treat a missing CSV as a reason to weaken eval assertions. Do not commit generated CSV output as ordinary source content.

Stale Eval State

Symptom:

  • eval_summaries.csv shows older totals than docs/demo_report.md.
  • A project data/eval_runtime_state.json was generated before the current fixtures changed.
  • The CSV says all cases pass, but a narrower eval command now fails.

Wrong fix:

  • Edit eval_runtime_state.json.
  • Edit CSV counts until they match the README.
  • Commit generated state as if it were a fixture.

Safe fix:

python -B scripts/dev.py evals
python -B scripts/dev.py eval-csv
python -B scripts/dev.py claims

If the regenerated CSV differs, review the fixture or logic change that caused it. Keep eval_summaries.csv untracked unless the release process explicitly asks for it.

Changed Case IDs

Symptom:

  • CSV totals change after renaming eval ids.
  • docs/demo_report.md or release notes mention a case id that no longer exists.
  • A reviewer cannot map a CSV count back to a specific regression case.

Wrong fix:

  • Rename eval ids only to make the CSV look cleaner.
  • Leave old case ids in documentation because total counts still match.

Safe fix:

python -B scripts/dev.py scenario-data
python -B scripts/dev.py evals
python -B scripts/dev.py claims

Update docs only after confirming the renamed case still protects the same invariant. Prefer adding a new eval id when the behavior changed meaningfully.

Unsafe Failure Counts

Symptom:

  • unsafe_failures is greater than 0 for any row.
  • docs/demo_report.md still claims unsafe failure counts are zero.
  • python -B scripts/dev.py claims fails after eval CSV generation.

Wrong fix:

  • Filter unsafe failures out of the CSV.
  • Rename unsafe metric keys so _unsafe_count ignores them.
  • Lower the README claim without fixing the regression.

Safe fix:

python -B scripts/dev.py evals
python -B scripts/dev.py claims
python -B scripts/dev.py safety

Treat any unsafe failure count as a release blocker. Fix the permission, approval, or release-blocking behavior first, then regenerate the CSV.

Generated Artifact Handling

Symptom:

  • eval_summaries.csv appears in git status.
  • A pull request includes generated CSV rows, eval_runtime_state.json, or runtime_state.json.
  • CSV rows include local paths, real identifiers, or copied external data.

Wrong fix:

  • Remove eval_summaries*.csv from .gitignore.
  • Commit generated runtime state to explain a local run.
  • Paste private or external evidence into the CSV.

Safe fix:

git diff --stat
git diff --check
git status --short --branch
python -B scripts/dev.py safety

Keep generated files out of commits unless the release process explicitly asks for source-visible evidence and the diff has been reviewed.

Review Checklist

  • python -B scripts/dev.py eval-csv wrote 3 rows.
  • Each row maps to a real project.
  • unsafe_failures is 0 for every project.
  • python -B scripts/dev.py claims agrees with the README and docs/demo_report.md.
  • eval_summaries.csv is not committed as ordinary source content.
  • No */data/eval_runtime_state.json or */data/runtime_state.json files are committed.
  • Any release-visible CSV artifact is regenerated from the intended commit and reviewed for private paths or real data.