As a PDS Node Operator, I want to track historical counts of missing and staged products over time

[jordanpadams]

Checked for duplicates

Yes - I've already checked

🧑‍🔬 User Persona(s)

PDS Node Operator, PDS Engineering Node

💪 Motivation

...so that I can monitor migration progress and generate burndown charts showing how the number of missing/staged products decreases over time as products are ingested into the new registry.

Relates to #476.

📖 Additional Details

The generate_registry_status_reports.py script currently overwrites the CSV reports on every run, discarding all historical data. To support burndown tracking, a time-series history file should be appended to on each run.

Proposed approach: append a dated row to docs/status/counts_history.csv on each run with columns: date, missing_bundles_total, missing_bundles_latest, missing_bundles_superseded, missing_collections_total, missing_collections_latest, missing_collections_superseded, staged_bundles_total, staged_collections_total.

The latest/superseded split is already being generated as of the work in #476.

Acceptance Criteria:

• generate_registry_status_reports.py appends one row per run to docs/status/counts_history.csv with a UTC datestamp and per-category counts (latest/superseded/total)
• History CSV is committed/pushed along with the other report files on each automated run
• docs/status/README.md documents the history file format and intended use for burndown tracking
• Counts tracked for: missing bundles (total, latest, superseded), missing collections (total, latest, superseded), staged bundles (total), staged collections (total)
• (Stretch) A burndown visualization or trend table is generated in the README from the history file

Notes:

• The history file must never be overwritten, only appended, so trend data accumulates over time
• Node-level breakdowns could be added in a second phase if needed

For Internal Dev Team To Complete

Acceptance Criteria

Given an execution of the report generation software
When I view docs/status/counts_history.csv
Then I expect a new row with the current UTC date and counts for all product categories

⚙️ Engineering Details

🎉 I&T