Skip to content

Commit bf60283

Browse files
jordanpadamsjordanpadams
authored
Merge pull request #461 from NASA-PDS/i460
Add script and config for generating registry status reports
2 parents ca2cb40 + 299276e commit bf60283

12 files changed

Lines changed: 6299 additions & 0 deletions

‎README.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,13 +19,43 @@ This repository specifically contains these PDS registry application resources:
1919
- the source for the user/administrator documentation, see `docs` folder
2020
- docker compose script to start/test the full application with its required components, see https://github.com/NASA-PDS/registry/blob/main/docker/README.md for details. This also includes integration tests (in `docker/postman`). The integration test datasets are maintained in NASA-PDS/registry-ref-data repository.
2121
- **For developers:** See [Integration Testing Guide](https://nasa-pds.github.io/registry/developer/integration-testing.html) for instructions on adding and running integration tests.
22+
- utility scripts for registry management and reporting, see `scripts` folder
2223

2324

2425
## Code of Conduct
2526

2627
All users and developers of the NASA-PDS software are expected to abide by our [Code of Conduct](https://github.com/NASA-PDS/.github/blob/main/CODE_OF_CONDUCT.md). Please read this to ensure you understand the expectations of our community.
2728

2829

30+
## Scripts
31+
32+
### Registry Status Report Generator
33+
34+
Generate CSV reports for missing and staged products in the PDS Registry:
35+
36+
```bash
37+
# Generate reports and commit/push to GitHub (default)
38+
./scripts/generate_registry_status_reports.py
39+
40+
# Generate reports only, without committing
41+
./scripts/generate_registry_status_reports.py --no-commit
42+
```
43+
44+
**Purpose:** Generates reports for missing and staged products in the registry.
45+
46+
**Output:** Creates four CSV files in `docs/status/`:
47+
- `missing_bundles_in_registry.csv` - Missing Product_Bundle records
48+
- `missing_collections_in_registry.csv` - Missing Product_Collection records
49+
- `staged_bundles_in_registry.csv` - Staged Product_Bundle records
50+
- `staged_collections_in_registry.csv` - Staged Product_Collection records
51+
52+
By default, the script commits and pushes these files to GitHub. Use `--no-commit` to disable this.
53+
54+
**Requirements:**
55+
- Python 3.12 or higher
56+
- `pds-registry-client` and PDS Registry credentials
57+
- See `scripts/README.md` for full configuration details
58+
2959
## Utilities
3060
* Treks
3161
* To deploy the package run one of these commands from the root directory:

‎conf/status/missing_bundles_per_node.json‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
2+
{
3+
"_source": ["node", "lidvid", "product_class"],
4+
"size": 2000,
5+
"query": {
6+
"bool": {
7+
"filter": [
8+
{
9+
"match_phrase": {
10+
"product_class": "Product_Bundle"
11+
}
12+
},
13+
{
14+
"match_phrase": {
15+
"found_in_registry": "false"
16+
}
17+
}
18+
]
19+
}
20+
}
21+
}

‎conf/status/missing_collections_per_node.json‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
2+
{
3+
"_source": ["node", "lidvid", "product_class"],
4+
"size": 2000,
5+
"query": {
6+
"bool": {
7+
"filter": [
8+
{
9+
"match_phrase": {
10+
"product_class": "Product_Collection"
11+
}
12+
},
13+
{
14+
"match_phrase": {
15+
"found_in_registry": "false"
16+
}
17+
}
18+
]
19+
}
20+
}
21+
}

‎conf/status/staged_bundles_per_node.json‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
2+
{
3+
"_source": ["ops:Harvest_Info/ops:node_name", "lidvid", "product_class", "ops:Harvest_Info/ops:harvest_date_time"],
4+
"size": 2000,
5+
"query": {
6+
"bool": {
7+
"filter": [
8+
{
9+
"match_phrase": {
10+
"product_class": "Product_Bundle"
11+
}
12+
},
13+
{
14+
"match_phrase": {
15+
"ops:Tracking_Meta/ops:archive_status": "staged"
16+
}
17+
}
18+
]
19+
}
20+
}
21+
}

‎conf/status/staged_collections_per_node.json‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
2+
{
3+
"_source": ["ops:Harvest_Info/ops:node_name", "lidvid", "product_class", "ops:Harvest_Info/ops:harvest_date_time"],
4+
"size": 2000,
5+
"query": {
6+
"bool": {
7+
"filter": [
8+
{
9+
"match_phrase": {
10+
"product_class": "Product_Collection"
11+
}
12+
},
13+
{
14+
"match_phrase": {
15+
"ops:Tracking_Meta/ops:archive_status": "staged"
16+
}
17+
}
18+
]
19+
}
20+
}
21+
}

‎docs/status/README.md‎

Lines changed: 188 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,188 @@
1+
# PDS Registry Status Reports
2+
3+
This directory contains automatically generated CSV reports that track the status of products in the PDS Registry.
4+
5+
## Metrics Summary
6+
7+
<!-- METRICS_START -->
8+
*Last updated: 2025-11-21 22:50:21 UTC*
9+
10+
### Missing Products by Node
11+
12+
| Node | Bundles | Collections |
13+
|------|---------|-------------|
14+
| KPDS | 1 | 2 |
15+
| PDS_ATM | 22 | 93 |
16+
| PDS_ENG | 7 | 7 |
17+
| PDS_GEO | 36 | 252 |
18+
| PDS_IMG | 60 | 876 |
19+
| PDS_PPI | 199 | 653 |
20+
| PDS_SBN | 72 | 117 |
21+
| **Total** | **397** | **2000** |
22+
23+
### Staged Products by Node
24+
25+
| Node | Bundles | Collections |
26+
|------|---------|-------------|
27+
| PDS_ATM | 3 | 6 |
28+
| PDS_GEO | 0 | 11 |
29+
| PDS_IMG | 13 | 38 |
30+
| PDS_NAIF | 0 | 63 |
31+
| PDS_PPI | 8 | 362 |
32+
| PDS_SBN | 35 | 57 |
33+
| PSA | 902 | 1463 |
34+
| **Total** | **961** | **2000** |
35+
36+
<!-- METRICS_END -->
37+
38+
## Reports
39+
40+
### Missing Products
41+
42+
These reports identify products that are marked as missing in the registry (`found_in_registry: false`):
43+
44+
- **`missing_bundles_in_registry.csv`** - Missing Product_Bundle records
45+
- **`missing_collections_in_registry.csv`** - Missing Product_Collection records
46+
47+
**CSV Format:** `NODE_ID, LIDVID, PRODUCT_CLASS`
48+
49+
**Example:**
50+
```
51+
"PDS_PPI","urn:nasa:pds:maven.rose.raw::1.21","Product_Bundle"
52+
"PDS_ENG","urn:nasa:pds:context::1.2","Product_Bundle"
53+
```
54+
55+
### Staged Products
56+
57+
These reports identify products that have an archive status of "staged" in the registry:
58+
59+
- **`staged_bundles_in_registry.csv`** - Staged Product_Bundle records
60+
- **`staged_collections_in_registry.csv`** - Staged Product_Collection records
61+
62+
**CSV Format:** `NODE_ID, LIDVID, PRODUCT_CLASS, HARVEST_DATE_TIME`
63+
64+
**Example:**
65+
```
66+
"PDS_SBN","urn:nasa:pds:bopps2014::1.0","Product_Bundle","2025-07-22T22:48:09.852492089Z"
67+
"PDS_ATM","urn:nasa:pds:insight_rad::1.0","Product_Bundle","2024-03-15T10:30:00Z"
68+
```
69+
70+
## How to Use These Files
71+
72+
### Viewing in GitHub
73+
74+
1. Navigate to this directory in GitHub: `docs/status/`
75+
2. Click on any CSV file to view it
76+
3. GitHub will render the CSV as a table for easy viewing
77+
78+
### Downloading Files
79+
80+
**Option 1: Download individual file from GitHub**
81+
1. Click on the CSV file in GitHub
82+
2. Click the "Raw" button
83+
3. Right-click and select "Save As..."
84+
85+
**Option 2: Download via command line**
86+
```bash
87+
# Download a specific report
88+
curl -O https://raw.githubusercontent.com/NASA-PDS/registry/main/docs/status/missing_bundles_in_registry.csv
89+
90+
# Download all reports
91+
cd docs/status
92+
for file in missing_bundles_in_registry.csv missing_collections_in_registry.csv \
93+
staged_bundles_in_registry.csv staged_collections_in_registry.csv; do
94+
curl -O https://raw.githubusercontent.com/NASA-PDS/registry/main/docs/status/$file
95+
done
96+
```
97+
98+
**Option 3: Clone the repository**
99+
```bash
100+
git clone https://github.com/NASA-PDS/registry.git
101+
cd registry/docs/status/
102+
```
103+
104+
### Opening Files
105+
106+
**Spreadsheet Applications:**
107+
- Microsoft Excel: File → Open → Select CSV file
108+
- Google Sheets: File → Import → Upload CSV file
109+
- LibreOffice Calc: File → Open → Select CSV file
110+
111+
**Text Editors:**
112+
- Any text editor (VS Code, Sublime Text, nano, vim, etc.)
113+
- CSV files are plain text and can be opened directly
114+
115+
**Command Line:**
116+
```bash
117+
# View entire file
118+
cat missing_bundles_in_registry.csv
119+
120+
# View first 10 lines
121+
head -10 missing_bundles_in_registry.csv
122+
123+
# Count records
124+
wc -l missing_bundles_in_registry.csv
125+
126+
# Search for specific node
127+
grep "PDS_SBN" staged_bundles_in_registry.csv
128+
129+
# Use column for formatted viewing
130+
column -t -s',' missing_bundles_in_registry.csv | less
131+
```
132+
133+
**Python:**
134+
```python
135+
import csv
136+
137+
# Read CSV file
138+
with open('missing_bundles_in_registry.csv', 'r') as f:
139+
reader = csv.reader(f)
140+
for row in reader:
141+
node, lidvid, product_class = row
142+
print(f"{node}: {lidvid}")
143+
144+
# Or use pandas for analysis
145+
import pandas as pd
146+
df = pd.read_csv('missing_bundles_in_registry.csv', names=['node', 'lidvid', 'product_class'])
147+
print(df.groupby('node').size())
148+
```
149+
150+
## Report Generation
151+
152+
These reports are automatically generated by the `generate_registry_status_reports.py` script located in the `scripts/` directory.
153+
154+
### Run Manually
155+
156+
```bash
157+
# From repository root
158+
./scripts/generate_registry_status_reports.py --no-commit
159+
```
160+
161+
See the [scripts/README.md](../../scripts/README.md) for more details on running the report generator.
162+
163+
## Data Source
164+
165+
- **Missing products** are queried from the `/en-legacy-registry/_search` endpoint
166+
- **Staged products** are queried from the `/*-registry/_search` endpoint (all registry indices)
167+
168+
Both queries return up to 2000 results per product type.
169+
170+
## Field Descriptions
171+
172+
- **NODE_ID** - The PDS node responsible for the product (e.g., PDS_SBN, PDS_ENG, PDS_GEO)
173+
- **LIDVID** - Logical Identifier with Version ID in the format `urn:nasa:pds:bundle_name::version`
174+
- **PRODUCT_CLASS** - Type of product (Product_Bundle or Product_Collection)
175+
- **HARVEST_DATE_TIME** - Timestamp when the product was harvested into the registry (ISO 8601 format, UTC)
176+
177+
## Questions or Issues
178+
179+
For questions about:
180+
- **Report contents or data**: Contact the PDS Engineering Node
181+
- **Script errors or enhancements**: Create an issue at [NASA-PDS/registry](https://github.com/NASA-PDS/registry/issues)
182+
- **Registry access**: Contact PDS operations team
183+
184+
## Related Documentation
185+
186+
- [Registry User Documentation](https://nasa-pds.github.io/registry/)
187+
- [Script Documentation](../../scripts/README.md)
188+
- [Main Repository README](../../README.md)

0 commit comments

Comments
 (0)