|
| 1 | +# CLAUDE.md |
| 2 | + |
| 3 | +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. |
| 4 | + |
| 5 | +## Project Overview |
| 6 | + |
| 7 | +This is the NASA PDS (Planetary Data System) Registry application repository. It's an **umbrella repository** that integrates multiple sub-components managed in their own repositories: |
| 8 | + |
| 9 | +- **registry-loader**: Tools to load PDS4 products to the registry |
| 10 | +- **opensearch**: Data store and search engine backend |
| 11 | +- **registry-sweepers**: Suite of scripts consolidating PDS4 product descriptions in OpenSearch |
| 12 | +- **registry-api**: PDS Search API service |
| 13 | +- **registry-ref-data**: Reference datasets for application integration tests |
| 14 | + |
| 15 | +The repository contains: |
| 16 | +- User/administrator documentation (in `docs/`) |
| 17 | +- Docker compose scripts for running the full application stack |
| 18 | +- Integration tests (in `docker/postman`) |
| 19 | +- Python utility scripts for Treks and GeoSTAC-Lola |
| 20 | + |
| 21 | +## Development Setup |
| 22 | + |
| 23 | +### Installation |
| 24 | + |
| 25 | +Install in editable mode with dev dependencies: |
| 26 | +```bash |
| 27 | +pip install -e '.[dev]' |
| 28 | +``` |
| 29 | + |
| 30 | +### Pre-commit Hooks |
| 31 | + |
| 32 | +Configure pre-commit hooks (runs on commit and push): |
| 33 | +```bash |
| 34 | +pre-commit install |
| 35 | +pre-commit install -t pre-push |
| 36 | +pre-commit install -t prepare-commit-msg |
| 37 | +pre-commit install -t commit-msg |
| 38 | +``` |
| 39 | + |
| 40 | +The hooks check for: |
| 41 | +- Secrets detection |
| 42 | +- Code formatting |
| 43 | +- PEP8 compliance via flake8 |
| 44 | +- Type hints via mypy |
| 45 | +- Import ordering |
| 46 | +- Test execution (on push) |
| 47 | + |
| 48 | +## Common Development Commands |
| 49 | + |
| 50 | +### Testing |
| 51 | + |
| 52 | +Run all tests with pytest (uses parallel execution by default): |
| 53 | +```bash |
| 54 | +pytest |
| 55 | +``` |
| 56 | + |
| 57 | +Run tests with coverage: |
| 58 | +```bash |
| 59 | +pytest --cov=src --cov-report=xml |
| 60 | +``` |
| 61 | + |
| 62 | +Run a single test file: |
| 63 | +```bash |
| 64 | +pytest src/pds/registry/tests/test_true.py |
| 65 | +``` |
| 66 | + |
| 67 | +### Linting |
| 68 | + |
| 69 | +Run all linters via tox: |
| 70 | +```bash |
| 71 | +tox -e lint |
| 72 | +``` |
| 73 | + |
| 74 | +Or run individual linters: |
| 75 | +```bash |
| 76 | +flake8 src |
| 77 | +mypy src |
| 78 | +``` |
| 79 | + |
| 80 | +### Testing with Tox |
| 81 | + |
| 82 | +Run tests in Python 3.13 environment: |
| 83 | +```bash |
| 84 | +tox -e py313 |
| 85 | +``` |
| 86 | + |
| 87 | +Run all tox environments (tests, docs, lint): |
| 88 | +```bash |
| 89 | +tox |
| 90 | +``` |
| 91 | + |
| 92 | +### Documentation |
| 93 | + |
| 94 | +Build Sphinx documentation: |
| 95 | +```bash |
| 96 | +cd docs |
| 97 | +make html |
| 98 | +``` |
| 99 | + |
| 100 | +Generated documentation will be in `docs/build/html/`. |
| 101 | + |
| 102 | +Alternatively, build docs via tox: |
| 103 | +```bash |
| 104 | +tox -e docs |
| 105 | +``` |
| 106 | + |
| 107 | +### Docker Development |
| 108 | + |
| 109 | +The `docker/` directory contains the full application stack. Key profiles: |
| 110 | + |
| 111 | +**Start backend services (Elasticsearch + Registry API):** |
| 112 | +```bash |
| 113 | +cd docker |
| 114 | +docker compose --profile=pds-core-registry up --detach |
| 115 | +``` |
| 116 | + |
| 117 | +**Start development environment with test data (no API):** |
| 118 | +```bash |
| 119 | +docker compose --profile=dev-api up |
| 120 | +``` |
| 121 | + |
| 122 | +**Run integration tests with test data:** |
| 123 | +```bash |
| 124 | +docker compose --profile=int-registry-batch-loader up |
| 125 | +``` |
| 126 | + |
| 127 | +**Clean up deployment:** |
| 128 | +```bash |
| 129 | +docker compose --profile=int-registry-batch-loader down --volumes |
| 130 | +``` |
| 131 | + |
| 132 | +**Note:** Before first run, generate certificates: |
| 133 | +```bash |
| 134 | +cd docker/certs |
| 135 | +./generate-certs.sh |
| 136 | +``` |
| 137 | + |
| 138 | +## Architecture Notes |
| 139 | + |
| 140 | +### Python Package Structure |
| 141 | + |
| 142 | +- **Namespace package**: Uses `pds` namespace with `pds.registry` package |
| 143 | +- **Source layout**: Code is in `src/` directory (NOT flat layout) |
| 144 | +- **Version management**: Version stored in `src/pds/registry/VERSION.txt` |
| 145 | +- **Entry points**: Two CLI tools are defined: |
| 146 | + - `create-treks-pds4`: Creates PDS4 labels for Treks API layers |
| 147 | + - `create-lola-pds4`: Creates PDS4 labels for Lola point clouds |
| 148 | + |
| 149 | +### Utilities |
| 150 | + |
| 151 | +Two utility modules exist under `src/pds/registry/utils/`: |
| 152 | +1. **treks**: Generate PDS4 labels from Treks API data |
| 153 | +2. **geostac**: Generate PDS4 labels for Lola GeoSTAC data (requires LOLA GDR data loaded in registry) |
| 154 | + |
| 155 | +### Docker Compose Architecture |
| 156 | + |
| 157 | +The docker-compose.yml defines multiple profiles for different deployment scenarios: |
| 158 | +- Components communicate via a Docker network named `pds` |
| 159 | +- Elasticsearch/OpenSearch runs on port 9200 |
| 160 | +- Registry API published on port 8080 (HTTP) and 8443 (HTTPS with self-signed cert) |
| 161 | +- Configuration files in `docker/default-config/` |
| 162 | +- Environment variables in `docker/.env` |
| 163 | + |
| 164 | +### Integration Testing |
| 165 | + |
| 166 | +Integration tests use Postman collections located in `docker/postman/`. These tests verify the full stack including data loading and API queries. |
| 167 | + |
| 168 | +**For detailed instructions on running and adding integration tests, see [Integration Testing Guide](https://nasa-pds.github.io/registry/developer/integration-testing.html) or [docs/source/developer/integration-testing.rst](docs/source/developer/integration-testing.rst).** |
| 169 | + |
| 170 | +Key points: |
| 171 | +- Tests run automatically on feature branches and main branch via GitHub Actions |
| 172 | +- To add a test: update TestRail, create Postman request with TestRail ID in assertions, export collection |
| 173 | +- Run tests locally: `docker compose --profile=int-registry-batch-loader up` |
| 174 | + |
| 175 | +## Code Quality Standards |
| 176 | + |
| 177 | +- **Python version**: Requires Python 3.13+ |
| 178 | +- **Line length**: 120 characters (configured in setup.cfg and pyproject.toml) |
| 179 | +- **Type hints**: Required, checked by mypy |
| 180 | +- **Docstring convention**: Google style |
| 181 | +- **Import ordering**: Managed by reorder_python_imports pre-commit hook |
| 182 | +- **Coverage**: Tests run with coverage reporting, omitting `__init__.py` and `_version.py` |
| 183 | + |
| 184 | +## CI/CD |
| 185 | + |
| 186 | +Two main workflows: |
| 187 | +- **unstable-cicd**: Runs on push to `main`, creates SNAPSHOT releases |
| 188 | +- **stable-cicd**: Runs on push to `release/<version>` branches |
| 189 | + |
| 190 | +Both use the NASA-PDS Roundup action for building and releasing. |
| 191 | + |
| 192 | +### Registry Status Reporting |
| 193 | + |
| 194 | +`scripts/generate_registry_status_reports.py` — runs on a schedule to generate CSV reports in `docs/status/` tracking missing and staged products. Key details: |
| 195 | + |
| 196 | +- Requires AWS/Cognito credentials via `~/.pds/.registry-client` or `.env` |
| 197 | +- Uses `pds-registry-client` from the **same venv as the running Python** (resolved via `sys.executable`); do not rely on shell PATH |
| 198 | +- Queries `conf/status/*.json` OpenSearch DSL files against the legacy and current registry indices |
| 199 | +- For missing products, generates one CSV per type with a `superseded` column (`true`/`false`) indicating whether a LIDVID is the latest version for its LID or an older version |
| 200 | +- Appends one row to `docs/status/counts_history.csv` on every run for burndown tracking — this file is **append-only, never overwritten** |
| 201 | +- Run with `--no-commit` to generate locally without pushing |
| 202 | + |
| 203 | +`scripts/backfill_history.py` — one-off utility to populate `counts_history.csv` from git history of the status CSVs. Safe to re-run (skips dates already present). |
| 204 | + |
| 205 | +```bash |
| 206 | +python scripts/backfill_history.py --dry-run # preview |
| 207 | +python scripts/backfill_history.py # write |
| 208 | +``` |
| 209 | + |
| 210 | +## Important Notes |
| 211 | + |
| 212 | +- **Secrets detection** is currently disabled in pre-commit (see comments in `.pre-commit-config.yaml`) |
| 213 | +- **Black formatter** is disabled (see comments in `.pre-commit-config.yaml`) |
| 214 | +- The repository requires several sub-component Docker images to be available for full integration testing |
| 215 | +- Default Docker configurations use default passwords - **DO NOT use in production** |
| 216 | +- Test data is downloaded from `https://pds-gamma.jpl.nasa.gov/` during integration tests |
0 commit comments