Skip to content

Commit 369d3cc

Browse files
jordanpadamsclaude
jordanpadams
and
claude
committed
Split missing products reports by latest/superseded version and add burndown tracking
- generate_registry_status_reports.py: for missing bundles/collections, produce three CSVs per type (overall, *_latest_*, *_superseded_*) by grouping LIDVIDs by LID and comparing versions numerically so 3.9 < 3.13. Resolves pds-registry-client via sys.executable rather than shell PATH to avoid venv activation issues. Appends one row per run to docs/status/counts_history.csv for burndown chart tracking (append-only, never overwritten). - backfill_history.py: new one-off script to populate counts_history.csv from git history of the status CSVs; idempotent (skips dates already present). - docs/status/counts_history.csv: initial 5-snapshot backfill from git history (2025-11-21 through 2026-03-18). - docs/status/README.md: updated metrics table to show Latest/Superseded/Total columns; added Historical Counts section documenting the burndown CSV format. - .gitignore: ignore *.code-workspace files. - CLAUDE.md: document the status reporting scripts and operational notes. Closes #481. Relates to #476. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent d1df816 commit 369d3cc

14 files changed

Lines changed: 8262 additions & 2892 deletions

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,3 +106,6 @@ venv/
106106
treks_xml/
107107
lola_xml/
108108
create-treks-pds4-log.log
109+
110+
# IDE workspace files
111+
*.code-workspace

‎CLAUDE.md‎

Lines changed: 216 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,216 @@
1+
# CLAUDE.md
2+
3+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4+
5+
## Project Overview
6+
7+
This is the NASA PDS (Planetary Data System) Registry application repository. It's an **umbrella repository** that integrates multiple sub-components managed in their own repositories:
8+
9+
- **registry-loader**: Tools to load PDS4 products to the registry
10+
- **opensearch**: Data store and search engine backend
11+
- **registry-sweepers**: Suite of scripts consolidating PDS4 product descriptions in OpenSearch
12+
- **registry-api**: PDS Search API service
13+
- **registry-ref-data**: Reference datasets for application integration tests
14+
15+
The repository contains:
16+
- User/administrator documentation (in `docs/`)
17+
- Docker compose scripts for running the full application stack
18+
- Integration tests (in `docker/postman`)
19+
- Python utility scripts for Treks and GeoSTAC-Lola
20+
21+
## Development Setup
22+
23+
### Installation
24+
25+
Install in editable mode with dev dependencies:
26+
```bash
27+
pip install -e '.[dev]'
28+
```
29+
30+
### Pre-commit Hooks
31+
32+
Configure pre-commit hooks (runs on commit and push):
33+
```bash
34+
pre-commit install
35+
pre-commit install -t pre-push
36+
pre-commit install -t prepare-commit-msg
37+
pre-commit install -t commit-msg
38+
```
39+
40+
The hooks check for:
41+
- Secrets detection
42+
- Code formatting
43+
- PEP8 compliance via flake8
44+
- Type hints via mypy
45+
- Import ordering
46+
- Test execution (on push)
47+
48+
## Common Development Commands
49+
50+
### Testing
51+
52+
Run all tests with pytest (uses parallel execution by default):
53+
```bash
54+
pytest
55+
```
56+
57+
Run tests with coverage:
58+
```bash
59+
pytest --cov=src --cov-report=xml
60+
```
61+
62+
Run a single test file:
63+
```bash
64+
pytest src/pds/registry/tests/test_true.py
65+
```
66+
67+
### Linting
68+
69+
Run all linters via tox:
70+
```bash
71+
tox -e lint
72+
```
73+
74+
Or run individual linters:
75+
```bash
76+
flake8 src
77+
mypy src
78+
```
79+
80+
### Testing with Tox
81+
82+
Run tests in Python 3.13 environment:
83+
```bash
84+
tox -e py313
85+
```
86+
87+
Run all tox environments (tests, docs, lint):
88+
```bash
89+
tox
90+
```
91+
92+
### Documentation
93+
94+
Build Sphinx documentation:
95+
```bash
96+
cd docs
97+
make html
98+
```
99+
100+
Generated documentation will be in `docs/build/html/`.
101+
102+
Alternatively, build docs via tox:
103+
```bash
104+
tox -e docs
105+
```
106+
107+
### Docker Development
108+
109+
The `docker/` directory contains the full application stack. Key profiles:
110+
111+
**Start backend services (Elasticsearch + Registry API):**
112+
```bash
113+
cd docker
114+
docker compose --profile=pds-core-registry up -d
115+
```
116+
117+
**Start development environment with test data (no API):**
118+
```bash
119+
docker compose --profile=dev-api up
120+
```
121+
122+
**Run integration tests with test data:**
123+
```bash
124+
docker compose --profile=int-registry-batch-loader up
125+
```
126+
127+
**Clean up deployment:**
128+
```bash
129+
docker compose --profile=int-registry-batch-loader down --volume
130+
```
131+
132+
**Note:** Before first run, generate certificates:
133+
```bash
134+
cd docker/certs
135+
./generate-certs.sh
136+
```
137+
138+
## Architecture Notes
139+
140+
### Python Package Structure
141+
142+
- **Namespace package**: Uses `pds` namespace with `pds.registry` package
143+
- **Source layout**: Code is in `src/` directory (NOT flat layout)
144+
- **Version management**: Version stored in `src/pds/registry/VERSION.txt`
145+
- **Entry points**: Two CLI tools are defined:
146+
- `create-treks-pds4`: Creates PDS4 labels for Treks API layers
147+
- `create-lola-pds4`: Creates PDS4 labels for Lola point clouds
148+
149+
### Utilities
150+
151+
Two utility modules exist under `src/pds/registry/utils/`:
152+
1. **treks**: Generate PDS4 labels from Treks API data
153+
2. **geostac**: Generate PDS4 labels for Lola GeoSTAC data (requires LOLA GDR data loaded in registry)
154+
155+
### Docker Compose Architecture
156+
157+
The docker-compose.yml defines multiple profiles for different deployment scenarios:
158+
- Components communicate via a Docker network named `pds`
159+
- Elasticsearch/OpenSearch runs on port 9200
160+
- Registry API published on port 8080 (HTTP) and 8443 (HTTPS with self-signed cert)
161+
- Configuration files in `docker/default-config/`
162+
- Environment variables in `docker/.env`
163+
164+
### Integration Testing
165+
166+
Integration tests use Postman collections located in `docker/postman/`. These tests verify the full stack including data loading and API queries.
167+
168+
**For detailed instructions on running and adding integration tests, see [Integration Testing Guide](https://nasa-pds.github.io/registry/developer/integration-testing.html) or [docs/source/developer/integration-testing.rst](docs/source/developer/integration-testing.rst).**
169+
170+
Key points:
171+
- Tests run automatically on feature branches and main branch via GitHub Actions
172+
- To add a test: update TestRail, create Postman request with TestRail ID in assertions, export collection
173+
- Run tests locally: `docker compose --profile=int-registry-batch-loader up`
174+
175+
## Code Quality Standards
176+
177+
- **Python version**: Requires Python 3.13+
178+
- **Line length**: 120 characters (configured in setup.cfg and pyproject.toml)
179+
- **Type hints**: Required, checked by mypy
180+
- **Docstring convention**: Google style
181+
- **Import ordering**: Managed by reorder_python_imports pre-commit hook
182+
- **Coverage**: Tests run with coverage reporting, omitting `__init__.py` and `_version.py`
183+
184+
## CI/CD
185+
186+
Two main workflows:
187+
- **unstable-cicd**: Runs on push to `main`, creates SNAPSHOT releases
188+
- **stable-cicd**: Runs on push to `release/<version>` branches
189+
190+
Both use the NASA-PDS Roundup action for building and releasing.
191+
192+
### Registry Status Reporting
193+
194+
`scripts/generate_registry_status_reports.py` — runs on a schedule to generate CSV reports in `docs/status/` tracking missing and staged products. Key details:
195+
196+
- Requires AWS/Cognito credentials via `~/.pds/.registry-client` or `.env`
197+
- Uses `pds-registry-client` from the **same venv as the running Python** (resolved via `sys.executable`); do not rely on shell PATH
198+
- Queries `conf/status/*.json` OpenSearch DSL files against the legacy and current registry indices
199+
- For missing products, generates three CSVs per type: overall, `*_latest_*` (highest version per LID), `*_superseded_*` (older versions)
200+
- Appends one row to `docs/status/counts_history.csv` on every run for burndown tracking — this file is **append-only, never overwritten**
201+
- Run with `--no-commit` to generate locally without pushing
202+
203+
`scripts/backfill_history.py` — one-off utility to populate `counts_history.csv` from git history of the status CSVs. Safe to re-run (skips dates already present).
204+
205+
```bash
206+
python scripts/backfill_history.py --dry-run # preview
207+
python scripts/backfill_history.py # write
208+
```
209+
210+
## Important Notes
211+
212+
- **Secrets detection** is currently disabled in pre-commit (see comments in `.pre-commit-config.yaml`)
213+
- **Black formatter** is disabled (see comments in `.pre-commit-config.yaml`)
214+
- The repository requires several sub-component Docker images to be available for full integration testing
215+
- Default Docker configurations use default passwords - **DO NOT use in production**
216+
- Test data is downloaded from `https://pds-gamma.jpl.nasa.gov/` during integration tests

‎docs/status/README.md‎

Lines changed: 64 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -5,45 +5,52 @@ This directory contains automatically generated CSV reports that track the statu
55
## Metrics Summary
66

77
<!-- METRICS_START -->
8-
*Last updated: 2026-03-18 20:06:49 UTC*
8+
*Last updated: 2026-03-20 18:14:56 UTC*
99

1010
### Missing Products by Node
1111

12-
| Node | Bundles | Collections |
13-
|------|---------|-------------|
14-
| KPDS | 1 | 3 |
15-
| PDS_ATM | 23 | 201 |
16-
| PDS_ENG | 7 | 92 |
17-
| PDS_GEO | 37 | 537 |
18-
| PDS_IMG | 58 | 1806 |
19-
| PDS_PPI | 215 | 1412 |
20-
| PDS_SBN | 45 | 182 |
21-
| **Total** | **386** | **4233** |
12+
| Node | Latest Bundles | Superseded Bundles | Total Bundles | Latest Collections | Superseded Collections | Total Collections |
13+
|------|---------------:|-------------------:|--------------:|-------------------:|-----------------------:|------------------:|
14+
| KPDS | 1 | 0 | 1 | 3 | 0 | 3 |
15+
| PDS_ATM | 22 | 4 | 26 | 156 | 59 | 215 |
16+
| PDS_ENG | 2 | 6 | 8 | 18 | 75 | 93 |
17+
| PDS_GEO | 27 | 9 | 36 | 306 | 224 | 530 |
18+
| PDS_IMG | 40 | 24 | 64 | 1642 | 191 | 1833 |
19+
| PDS_PPI | 38 | 175 | 213 | 222 | 1211 | 1433 |
20+
| PDS_SBN | 23 | 21 | 44 | 116 | 63 | 179 |
21+
| **Total** | **153** | **239** | **392** | **2463** | **1823** | **4286** |
2222

2323
### Staged Products by Node
2424

2525
| Node | Bundles | Collections |
26-
|------|---------|-------------|
26+
|------|--------:|------------:|
2727
| PDS_ATM | 3 | 30 |
2828
| PDS_GEO | 0 | 16 |
29-
| PDS_IMG | 5 | 72 |
29+
| PDS_IMG | 0 | 64 |
3030
| PDS_NAIF | 0 | 183 |
31-
| PDS_PPI | 6 | 257 |
31+
| PDS_PPI | 1 | 274 |
3232
| PDS_RMS | 0 | 1 |
33-
| PDS_SBN | 26 | 147 |
33+
| PDS_SBN | 26 | 142 |
3434
| PSA | 902 | 4171 |
35-
| **Total** | **942** | **4877** |
35+
| **Total** | **932** | **4881** |
3636

3737
<!-- METRICS_END -->
3838

3939
## Reports
4040

4141
### Missing Products
4242

43-
These reports identify products that are marked as missing in the registry (`found_in_registry: false`):
43+
These reports identify products that are marked as missing in the registry (`found_in_registry: false`).
44+
Three variants are generated per product type by comparing version numbers numerically within each LID:
4445

45-
- **`missing_bundles_in_registry.csv`** - Missing Product_Bundle records
46-
- **`missing_collections_in_registry.csv`** - Missing Product_Collection records
46+
| File | Description |
47+
|------|-------------|
48+
| `missing_bundles_in_registry.csv` | All missing Product_Bundle records (overall) |
49+
| `missing_bundles_latest_in_registry.csv` | Only the highest-versioned missing bundle per LID |
50+
| `missing_bundles_superseded_in_registry.csv` | Older versions of missing bundles (superseded by a newer version) |
51+
| `missing_collections_in_registry.csv` | All missing Product_Collection records (overall) |
52+
| `missing_collections_latest_in_registry.csv` | Only the highest-versioned missing collection per LID |
53+
| `missing_collections_superseded_in_registry.csv` | Older versions of missing collections (superseded by a newer version) |
4754

4855
**CSV Format:** `NODE_ID, LIDVID, PRODUCT_CLASS`
4956

@@ -53,6 +60,35 @@ These reports identify products that are marked as missing in the registry (`fou
5360
"PDS_ENG","urn:nasa:pds:context::1.2","Product_Bundle"
5461
```
5562

63+
### Historical Counts (Burndown Tracking)
64+
65+
**`counts_history.csv`** — one row is appended per run; the file is **never overwritten** so data accumulates over time. Use this to plot a burndown of missing/staged products.
66+
67+
**CSV Format (header included):**
68+
```
69+
date,
70+
missing_bundles_total,missing_bundles_latest,missing_bundles_superseded,
71+
missing_collections_total,missing_collections_latest,missing_collections_superseded,
72+
staged_bundles_total,staged_collections_total
73+
```
74+
75+
**Example:**
76+
```
77+
date,missing_bundles_total,missing_bundles_latest,missing_bundles_superseded,missing_collections_total,missing_collections_latest,missing_collections_superseded,staged_bundles_total,staged_collections_total
78+
2026-03-20,386,250,136,4233,2100,2133,942,4877
79+
2026-03-21,380,245,135,4190,2080,2110,938,4850
80+
```
81+
82+
**Analyzing with Python/pandas:**
83+
```python
84+
import pandas as pd
85+
import matplotlib.pyplot as plt
86+
87+
df = pd.read_csv('counts_history.csv', parse_dates=['date'])
88+
df.plot(x='date', y=['missing_bundles_latest', 'missing_collections_latest'], title='Missing Products Burndown')
89+
plt.show()
90+
```
91+
5692
### Staged Products
5793

5894
These reports identify products that have an archive status of "staged" in the registry:
@@ -90,8 +126,15 @@ curl -O https://raw.githubusercontent.com/NASA-PDS/registry/main/docs/status/mis
90126

91127
# Download all reports
92128
cd docs/status
93-
for file in missing_bundles_in_registry.csv missing_collections_in_registry.csv \
94-
staged_bundles_in_registry.csv staged_collections_in_registry.csv; do
129+
for file in missing_bundles_in_registry.csv \
130+
missing_bundles_latest_in_registry.csv \
131+
missing_bundles_superseded_in_registry.csv \
132+
missing_collections_in_registry.csv \
133+
missing_collections_latest_in_registry.csv \
134+
missing_collections_superseded_in_registry.csv \
135+
staged_bundles_in_registry.csv \
136+
staged_collections_in_registry.csv \
137+
counts_history.csv; do
95138
curl -O https://raw.githubusercontent.com/NASA-PDS/registry/main/docs/status/$file
96139
done
97140
```

‎docs/status/counts_history.csv‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
date,missing_bundles_total,missing_bundles_latest,missing_bundles_superseded,missing_collections_total,missing_collections_latest,missing_collections_superseded,staged_bundles_total,staged_collections_total
2+
2025-11-21,397,197,200,2000,1353,647,961,2000
3+
2026-01-28,420,198,222,2000,1302,698,953,2000
4+
2026-03-03,420,198,222,2000,1302,698,953,2000
5+
2026-03-04,420,198,222,4231,2566,1665,952,5540
6+
2026-03-18,386,165,221,4233,2512,1721,942,4877
7+
2026-03-20,392,153,239,4286,2463,1823,932,4881

0 commit comments

Comments
 (0)