Add auto-generated human-readable Postman collection docs - #501
Conversation
- Add generate_collection_docs.py to convert postman_collection.json to Markdown with TOC, linked TestRail case IDs, and GitHub issue refs - Add generated postman_collection.md alongside the JSON source - Add GitHub Actions workflow to regenerate the doc on every push that touches postman_collection.json (resolves #497) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix TOC anchor mismatch: anchors now include HTTP method to match rendered headings - Add anchor deduplication for repeated request names - Remove unused extract_testrail_ids() function - Add encoding="utf-8" to open() and write_text() for deterministic output - Remove [skip ci] from workflow commit (no loop risk; trigger is .json not .md) - Regenerate postman_collection.md with corrected TOC anchors Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1280ba3 to
65d5a43
Compare
There was a problem hiding this comment.
Pull request overview
Adds automation to generate and commit a human-readable Markdown rendering of the Postman collection JSON to improve PR reviewability and traceability.
Changes:
- Added
docker/postman/generate_collection_docs.pyto convertpostman_collection.jsoninto a structured Markdown document (TOC, requests, Accept headers, tests, links). - Added a GitHub Actions workflow to regenerate and auto-commit
postman_collection.mdwhen the collection JSON changes on non-main branches. - Updated pre-commit configuration to exclude the generated Markdown from
end-of-file-fixer.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
docker/postman/postman_collection.md |
New auto-generated Markdown documentation output committed alongside the JSON. |
docker/postman/generate_collection_docs.py |
Script that parses the Postman collection JSON and renders Markdown with TOC, request details, and linkification. |
.github/workflows/postman-collection-docs.yml |
CI workflow to regenerate and commit the Markdown when the collection JSON changes. |
.pre-commit-config.yaml |
Excludes the generated Markdown file from the EOF fixer hook. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| def extract_test_names(test_script: str) -> list[tuple[str, list[str]]]: | ||
| """Return list of (test_name, [testrail_ids]) from pm.test() calls.""" | ||
| results = [] | ||
| for match in re.finditer(r'pm\.test\(\s*["\']([^"\']+)["\']', test_script): | ||
| name = match.group(1) | ||
| ids = re.findall(r"\bC\d{5,}\b", name) | ||
| # Strip leading IDs from the display name | ||
| display = re.sub(r"^(C\d+\s+)+", "", name).strip() | ||
| results.append((display, ids)) | ||
| return results |
There was a problem hiding this comment.
extract_test_names() only detects pm.test() calls where the first argument is wrapped in single/double quotes. The current collection uses template literals (backticks) and ${testrailId} interpolation (e.g., const testrailId = "C4440539"; + pm.test(`${testrailId} …`)), which results in generated docs containing literal ${testrailId} strings and missing TestRail links. Consider expanding parsing to handle backtick strings and, when ${testrailId} is used, extracting the testrailId constant from the same script block so the correct C####### link can be emitted.
| """Extract GitHub issue refs like NASA-PDS/registry-api#494 from a string.""" | ||
| return re.findall(r"NASA-PDS/[\w-]+#\d+", name) |
There was a problem hiding this comment.
extract_github_refs() only recognizes NASA-PDS/<repo>#<num> patterns. The collection also contains issue references in other common forms (e.g. NASA-PDS/registry-api/issues/66, and NASA-PDS/registry-api/#638), which are currently not converted into hyperlinks in the generated Markdown. Expanding the regex and link rendering to cover these variants would better meet the “GitHub issue references as hyperlinks” requirement.
| """Extract GitHub issue refs like NASA-PDS/registry-api#494 from a string.""" | |
| return re.findall(r"NASA-PDS/[\w-]+#\d+", name) | |
| """Extract GitHub issue refs and normalize them to NASA-PDS/<repo>#<num>.""" | |
| pattern = re.compile( | |
| r"(NASA-PDS/([\w-]+)(?:#(\d+)|/#(\d+)|/issues/(\d+)))" | |
| ) | |
| refs = [] | |
| for match in pattern.finditer(name): | |
| repo = match.group(2) | |
| issue_number = match.group(3) or match.group(4) or match.group(5) | |
| refs.append(f"NASA-PDS/{repo}#{issue_number}") | |
| return refs |
| def _build_anchor(heading_text: str, used_anchors: set[str]) -> str: | ||
| """Build a GitHub-style anchor from heading text, ensuring uniqueness.""" | ||
| anchor = re.sub(r"[^\w\s-]", "", heading_text.lower()).strip() | ||
| anchor = re.sub(r"[\s]+", "-", anchor) | ||
|
|
||
| if anchor in used_anchors: | ||
| suffix = 2 | ||
| unique_anchor = f"{anchor}-{suffix}" | ||
| while unique_anchor in used_anchors: | ||
| suffix += 1 | ||
| unique_anchor = f"{anchor}-{suffix}" | ||
| anchor = unique_anchor | ||
|
|
There was a problem hiding this comment.
_build_anchor() attempts to mimic GitHub’s duplicate-heading anchor scheme, but it starts suffixing duplicates at -2. GitHub generates #heading, then #heading-1, #heading-2, etc. If duplicate headings ever occur in the collection, the TOC links produced here will not match GitHub’s rendered anchors. Adjust the suffixing to start at 1 for the first duplicate to keep TOC links reliable.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
tloubrieu-jpl
left a comment
There was a problem hiding this comment.
Thanks @jordanpadams , that will help Catherine for the I&T
* Bump actions/checkout from 5 to 6 Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Bump actions/cache from 4 to 5 Bumps [actions/cache](https://github.com/actions/cache) from 4 to 5. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@v4...v5) --- updated-dependencies: - dependency-name: actions/cache dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Bump actions/upload-artifact from 5 to 6 Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 5 to 6. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v5...v6) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Bump SonarSource/sonarqube-scan-action from 6 to 7 Bumps [SonarSource/sonarqube-scan-action](https://github.com/sonarsource/sonarqube-scan-action) from 6 to 7. - [Release notes](https://github.com/sonarsource/sonarqube-scan-action/releases) - [Commits](SonarSource/sonarqube-scan-action@v6...v7) --- updated-dependencies: - dependency-name: SonarSource/sonarqube-scan-action dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * fix end of file * Update requirements * Update changelog * Commiting /github/workspace/src/pds/registry/VERSION.txt for stable release * Setting next dev version to 1.7.0 * switch to dev version of components * make integration tests work again * downgrade sweeper because latest stables builds are broken and do not work with the latest version of api * Update postman_collection.json (#470) * update registry testing First, modified .env and docker-compose to allow the user to move from a docker volume to a disk with more space. Second, added 8 tests for "exists" capability in the registry-api query language * Update postman_collection.json Change the way wildcarding is expressed. * Update postman_collection.json Put * back in because of too happy to delete. --------- Co-authored-by: Al Niessner <Al.Niessner@xxx.xxx> Co-authored-by: thomas loubrieu <thomas.loubrieu@jpl.nasa.gov> * fix wrong automated test * Update missing products reports - 2026-01-28 10:08:52 * Update README.md * Create requirements.txt * Update README.md * Update Integration test to account for the refactoring of the ancestry management (#474) * update properties count test * disable 2nd-order membership tests as this functionality is deprecated and must be reimplemented or removed at a later date * update properties count test again- this does not appear to be consistent and may be picking up metadata keys * use latest verion of sweeper after successful integration tests * Add AOSS vs. Managed OpenSearch Trade Study Refs #451 * Fix bug in requirements.txt * Update missing products reports - 2026-03-03 15:55:00 * Update missing products reports - 2026-03-04 13:55:14 * Update queries to include up to 10000 records Resolves #471 * Update missing products reports - 2026-03-04 13:57:11 * Bump actions/upload-artifact from 6 to 7 Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 6 to 7. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v6...v7) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Update missing products reports - 2026-03-18 12:06:51 * Update missing products reports - 2026-03-18 12:07:22 * Update missing products reports - 2026-03-18 12:13:30 * Update missing products reports - 2026-03-18 13:06:49 * Split missing products reports by latest/superseded version and add burndown tracking - generate_registry_status_reports.py: for missing bundles/collections, produce three CSVs per type (overall, *_latest_*, *_superseded_*) by grouping LIDVIDs by LID and comparing versions numerically so 3.9 < 3.13. Resolves pds-registry-client via sys.executable rather than shell PATH to avoid venv activation issues. Appends one row per run to docs/status/counts_history.csv for burndown chart tracking (append-only, never overwritten). - backfill_history.py: new one-off script to populate counts_history.csv from git history of the status CSVs; idempotent (skips dates already present). - docs/status/counts_history.csv: initial 5-snapshot backfill from git history (2025-11-21 through 2026-03-18). - docs/status/README.md: updated metrics table to show Latest/Superseded/Total columns; added Historical Counts section documenting the burndown CSV format. - .gitignore: ignore *.code-workspace files. - CLAUDE.md: document the status reporting scripts and operational notes. Closes #481. Relates to #476. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Add API validator script to spot-check missing products validate_missing_in_api.py reads the overall missing bundle and collection CSVs, randomly samples up to 500 LIDVIDs (configurable via --sample/--seed), queries https://pds.nasa.gov/api/search/1/products/{lidvid} for each, and reports HTTP status codes. Products returning 200 are flagged as unexpectedly present in the API. Features: - Concurrent requests via ThreadPoolExecutor (--workers, default 10) - Retry logic for transient errors (429, 5xx) - Results written to docs/status/validation_results.csv - --dry-run, --found-only, --sample 0 (full pool) modes - Fixed seed support for reproducible sampling Closes #484. Relates to #476. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Update scripts/generate_registry_status_reports.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Initial plan * Initial plan * Initial plan * Tighten type hints in split_by_version() in backfill_history.py Co-authored-by: jordanpadams <33492486+jordanpadams@users.noreply.github.com> Agent-Logs-Url: https://github.com/NASA-PDS/registry/sessions/ab42f1c3-5d84-40a5-b8c0-c00a2938c827 * Initial plan * Use --detach instead of -d in docker compose command in CLAUDE.md Co-authored-by: jordanpadams <33492486+jordanpadams@users.noreply.github.com> Agent-Logs-Url: https://github.com/NASA-PDS/registry/sessions/3ad3b701-4ef9-4645-af27-3b698844b1e7 * Compute CSV counts once and reuse in both generate_metrics_from_csvs and append_history_row Co-authored-by: jordanpadams <33492486+jordanpadams@users.noreply.github.com> Agent-Logs-Url: https://github.com/NASA-PDS/registry/sessions/00d36611-6202-4258-9b9c-e5fa636f42da * Fix --volume to --volumes in CLAUDE.md docker compose command Co-authored-by: jordanpadams <33492486+jordanpadams@users.noreply.github.com> Agent-Logs-Url: https://github.com/NASA-PDS/registry/sessions/42ce8b7b-f5ac-4e73-b3a6-b5f5fe6d9dc4 * Consolidate missing products CSVs into single file with superseded column Instead of generating three separate CSVs per product type (overall, latest, superseded), generate one CSV with a `superseded` column (true/false) indicating whether each LIDVID is the latest version for its LID. History counts and README metrics table are unchanged. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Update missing products reports - 2026-03-23 18:57:33 * Remove CSVs we are no longer generating * Update missing products reports - 2026-03-24 11:40:39 * Update missing products reports - 2026-03-24 12:06:47 * Update missing products reports - 2026-03-24 12:45:49 * Update missing products reports - 2026-03-24 13:54:53 * Update missing products reports - 2026-03-24 14:41:16 * Fix superseded version detection for missing bundles and collections Queries now fetch all products (not just found_in_registry=false) so that version ordering is determined across the full set. Python filters to missing rows after annotating superseded status, ensuring a missing LIDVID is correctly marked superseded when a higher version exists in the registry. Also adds CSV headers to output files and skips header rows in count aggregation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * skip broken tests * skip broken tests * down to a minor mismatch * minimal skips * Improve registry documentation for new users (#392, #391, #393, #394) (#502) This commit addresses multiple documentation issues to better support new PDS users: **Issue #391 - Add links to component documentation:** - Added link to registry-client external docs from Registry Client section - Added links from Registry Manager to update_status, delete_data, and create_reg pages - Added link from Harvest section to load1.html user guide **Issue #393 - Improve documentation consistency and ordering:** - Reordered components: Harvest → Registry Manager → Registry Client → OpenSearch → Registry API - Made Authentication/Authorization a subsection of OpenSearch - Renamed "API" to "Registry API (PDS Search API)" for clarity and consistency - Added note about archive status affecting API visibility - Clarified archive status values with descriptions: * archived - publicly visible * certified - publicly visible * restricted - not publicly visible * staged - default, not publicly available - Fixed inconsistent terminology: changed "uploaded" to "loaded" in load1.rst **Issue #394 - Update registry-manager documentation:** - Updated parameter documentation to reflect current CLI (2 required, 2 optional) - Replaced deprecated -es with -registry parameter - Updated examples to use new parameter names - Fixed "registered" to "certified" in Registry API section These changes improve discoverability and clarity for users new to the PDS Registry system. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * Add auto-generated human-readable Postman collection docs (#501) * Add human-readable Postman collection docs with auto-generation workflow - Add generate_collection_docs.py to convert postman_collection.json to Markdown with TOC, linked TestRail case IDs, and GitHub issue refs - Add generated postman_collection.md alongside the JSON source - Add GitHub Actions workflow to regenerate the doc on every push that touches postman_collection.json (resolves #497) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Exclude postman_collection.md from end-of-file-fixer hook Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Restrict postman docs workflow to feature branches only Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Replace third-party commit action with git CLI commands Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address Copilot review feedback on postman doc generator - Fix TOC anchor mismatch: anchors now include HTTP method to match rendered headings - Add anchor deduplication for repeated request names - Remove unused extract_testrail_ids() function - Add encoding="utf-8" to open() and write_text() for deterministic output - Remove [skip ci] from workflow commit (no loop risk; trigger is .json not .md) - Regenerate postman_collection.md with corrected TOC anchors Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: thomas loubrieu <60993872+tloubrieu-jpl@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * update exists operator tests as NASA-PDS/registry-api#741 * update test for updated members/members and member-of/member-of support * add missing host for elasticsearch service for docker compose to work with option --project-name * simplify the api launch * fix docker compose for github action support * use --project-name value in the docker compose through a new env variable * add missing numbering suffix * add healthcheck on api and use it as a dependency for other services to start. * simplify docker compose, adjust test or latest PR of registry-api #753 * change test expectation for GONE members/members end-point * add check for good error message * docs: regenerate postman_collection.md * non existance has changed * docs: regenerate postman_collection.md * fixup last test failure * docs: regenerate postman_collection.md * Update missing products reports - 2026-04-27 14:15:27 * Bump SonarSource/sonarqube-scan-action from 7 to 8 Bumps [SonarSource/sonarqube-scan-action](https://github.com/sonarsource/sonarqube-scan-action) from 7 to 8. - [Release notes](https://github.com/sonarsource/sonarqube-scan-action/releases) - [Commits](SonarSource/sonarqube-scan-action@v7...v8) --- updated-dependencies: - dependency-name: SonarSource/sonarqube-scan-action dependency-version: '8' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Initial import of terraform for provisioned opensearch * Correct committed versions of main and variable tf files. * Initial commit of index template tf scripts. * Fix opensearch provider. * Add pipeline terraform. * Make main provider.tf consistent. * Make index_templates provider.tf consistent. * Intermediate commit (to preserve content) of OSI pipeline tf scripts. * Switched strategy for osi pipeline config body - went to an external yaml file. * Fixed specification of policy file variable. * Update missing products reports - 2026-05-11 13:35:12 * Revise pipeline terraform scripts. * Add burnup charts, loaded product reports, and per-node tracking - Add burnup_chart.html with interactive time-series visualization - Add burnup_history.csv, burnup_by_node.csv and their *_latest variants - Add loaded_bundles/collections CSVs and per-node OpenSearch DSL queries - Update generate_registry_status_reports.py to produce all new reports - Update docs/status/README.md to document new outputs Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com> * Add workflow to publish burnup chart to GitHub Pages - New publish-burnup-chart.yml triggers on push to main when docs/status/burnup_chart.html changes, copying it to gh-pages/status/ - Add docs/source/status.rst with link to the live chart - Wire status.rst into the Sphinx toctree in index.rst Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com> * Add burnup chart publishing to GitHub Pages - Fix publish-burnup-chart workflow: save burnup_chart.html to /tmp before checking out gh-pages branch so the file survives the workspace replacement - Add docs/source/status.rst linking to the live chart - Wire status.rst into the Sphinx toctree - Add docs/status/burnup_chart.html to main so the workflow can trigger Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com> * Update publish-burnup-chart.yml * Update CLAUDE.md and docs/status/README with loaded product and burnup chart documentation - Expand Registry Status Reporting in CLAUDE.md: correct venv invocation, table of all six query config files with endpoints, data population distinction (legacy Solr vs new OpenSearch / PSA inflation), AOSS pagination caveat, full output file table, burnup chart details - Add "How Numbers Are Calculated" section to docs/status/README.md explaining missing/loaded/staged/burnup derivation and the superseded flag algorithm - Add Loaded Products and Burnup Charts report sections to README - Update download script with new loaded_* and burnup_* files Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com> * Update metrics * Fix publish workflow to checkout triggering branch instead of hardcoded main Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com> * Improve user documentation with procedural step-by-step instructions Rewrites connection-setup, installation, and all user task docs to use numbered steps throughout. Adds test_connection page, standardizes config file naming to registry-{tool}-config-{node}-{venue} pattern under ~/.pds/, and fixes autosectionlabel warnings in conf.py. Resolves #515 Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com> * Switch over to full block for index template settings to provide more flexibility * Fix typo for pipeline role arn. * Add index alias support. * Update missing products reports - 2026-06-09 11:21:52 * Expand registry loading status page with burnup chart and report links Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Scale x-axis tick density to selected date range on burnup chart Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix burnup chart x-axis to use calendar time scale independent of data Switch from category scale to Chart.js time scale with chartjs-adapter-date-fns. The axis now spans the full selected date range with evenly-spaced calendar ticks (weekly for ≤6mo, monthly for ≤2yr, yearly for all time) regardless of where data points happen to fall. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Update missing products reports - 2026-06-11 08:05:34 * Update changelog * Update reports * Update changelog * add test for security patch, update docker compose for latest sweeper docker upgrade * docs: regenerate postman_collection.md * update test after sweeper upgrade * add explicit pull_policy for registry-api to make sure the local image is used when available * Fix race condition: add depends_on between domain policy and domain Without this, Terraform may attempt to create the domain access policy before the OpenSearch domain exists, causing a ResourceNotFoundException. * Bump actions/checkout from 6 to 7 Bumps [actions/checkout](https://github.com/actions/checkout) from 6 to 7. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v6...v7) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Bump actions/cache from 5 to 6 Bumps [actions/cache](https://github.com/actions/cache) from 5 to 6. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@v5...v6) --- updated-dependencies: - dependency-name: actions/cache dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Add script to detect and delete stale LDD_Info sentinel records Adds scripts/detect_stale_ldd_sentinels.py which queries all node registry -dd indexes for LDD_Info sentinels that have zero field documents, reports them with ready-to-paste Dev Tools delete queries, and optionally deletes them with explicit confirmation. Key features: - Paginates sentinel fetch with search_after so all records are found - Checkpoints progress after each _count query; interrupted runs resume - Retries transient connection errors with exponential backoff - --delete flag required for any writes; prompts for 'yes' confirmation - --report / --reset / --checkpoint flags for operational control Fixes: NASA-PDS/registry-loader#89 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Re-enable secrets detection using standard detect-secrets Replaces the disabled slim-detect-secrets GitHub Action and commented-out pre-commit hook with the updated approach from cloud-tools: - New `scripts/detect_secrets_baseline.sh` with scan/audit/check modes, `.detect-secrets-ignore` support, and Python-based comparison (no jq) - Workflow delegates to the script instead of inlining logic - `.detect-secrets-ignore` excludes docker/default-config, docs, test data - `.secrets.baseline` regenerated with detect-secrets~=1.5.0; all 3 findings audited as false positives (Jenkins credential ref, test placeholder, md5 URL param) - Pre-commit hook re-enabled pointing to the new script Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix --reset flag skipping sentinel fetch load_checkpoint returned {"sentinels": [], "results": {}} as the default when no file exists, so after --reset deleted the checkpoint the reload saw "sentinels" was present and skipped the fetch entirely, printing "Resuming from checkpoint — 0 sentinel(s) already fetched" / "Nothing to do." Fix: return {"results": {}} so a fresh/reset checkpoint has no "sentinels" key and the fetch is always triggered. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix status report showing false 'not yet checked' warning print_status_report compared len(results) against len(sentinels) (raw hit count), but results deduplicates by (index, namespace) key. With 420 raw hits but only 164 unique pairs, the report falsely reported 256 unchecked. Fix: compute total from unique (index, ns) keys in the sentinels list so the denominator matches what results actually tracks. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Update secrets and dependencies * Fix linter issues * Update .gitignore * Add terraform script for complete deployment of the Registry in the Dev Account (#475) * initialize terraform to deploy simple regitry in dev, wip * wip: terraform deployment of opensearch serverless collection * make the depoyment work with external definition of IAM role in pds-cds-infra repository * adjust the role/policy management without using SSM for now * upgrade provider for compatibility purpose * modularize terraform, to make external policies stand out * add missing module directory * clean role argument names * rename opensearch api and dashboard access policy * wip: untested credentials endpoint for registry-tools in terraform * automate registry initialization with registry-mgr/harvest * add registry-manager set-archive-status command to match reference test setup * add usage documentation in readme. * Update missing products reports - 2026-03-18 10:01:42 * Update missing products reports - 2026-03-18 12:04:37 * add missing api gateway and first version of automated initialization with test data on AWS * add creation of the build directory * validate terraform apply/destroy * add build layer directory for plan phase. * use pip3 instead of pip which might not be available locally. * wip: registry deployed in dev with api and sweeper manually added from their own terraform, integration test still fail. * remove automated lias creation as it does not work yet * add comment for commented out code, specifically dedicated to Sean! * update to initialize node registry without ref data. * update to have log when the lambda layer is build as it does not work from terragrunt call * try to clarify what path module is * force to recreate the layer whenever the zip file has to be redeployed * minor fixes, logs * add sleep to make sure the harvest test products are properly indexed * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Fix COGNITO_ALLOWED_GROUPS substring matching by splitting into a set * Strip whitespace from COGNITO_ALLOWED_GROUPS entries after splitting * force delete registry indexes before deployment, make vpc endpoint service name unique to handle multiple deployment in the same account * update sweeper call arguments per later update on sweeper docker image * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * update docker compose to make the test wait for api to be ready * rename opensearch module as opensearch_serverless module to avoid conflicts * make it compliant with python 3.12 * rename opensearch directory into opensearch_managed for consistency with opensearch_serverless * fix renamed vpc endpoint service * add the vpc endpoint with name depending on the opensearch collection name * finally remove the collection service because it cannot work * add missing relay variable * Linter updates * fix obsolete README instruction --------- Co-authored-by: thomas loubrieu <thomas.loubrieu@jpl.nasa.gov> Co-authored-by: Thomas Loubrieu <loubrieu@jpl.nasa.gov> Co-authored-by: Jordan Padams <33492486+jordanpadams@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Jordan Padams <jordan.h.padams@jpl.nasa.gov> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Bump python-jose from 3.3.0 to 3.4.0 in /terraform/lambda/src/layer Bumps [python-jose](https://github.com/mpdavis/python-jose) from 3.3.0 to 3.4.0. - [Release notes](https://github.com/mpdavis/python-jose/releases) - [Changelog](https://github.com/mpdavis/python-jose/blob/master/CHANGELOG.md) - [Commits](mpdavis/python-jose@3.3.0...3.4.0) --- updated-dependencies: - dependency-name: python-jose dependency-version: 3.4.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Bump requests from 2.32.3 to 2.33.0 in /terraform/lambda/src/layer Bumps [requests](https://github.com/psf/requests) from 2.32.3 to 2.33.0. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.32.3...v2.33.0) --- updated-dependencies: - dependency-name: requests dependency-version: 2.33.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Update dependabot.yml * Update dependabot.yml * Update unstable-cicd.yaml * Update branch-cicd.yaml * Update missing products reports - 2026-07-13 13:56:58 * Fix linter issues * Bump actions/checkout in the github-actions group across 1 directory Bumps the github-actions group with 1 update in the / directory: [actions/checkout](https://github.com/actions/checkout). Updates `actions/checkout` from 4 to 7 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v4...v7) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions ... Signed-off-by: dependabot[bot] <support@github.com> * Update requirements * Update changelog * Update CLAUDE.md to require pre-commit install or tox before pushing Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Update requirements * Update changelog * update sweepers command to reflect ENTRYPOINT directive introduced in 2b83e392 * Update requirements * Update changelog * Add % completion by node charts and flip commit default to --no-commit - Add "% Completion by Node" section to burnup_chart.html with four normalized line charts (all-versions + latest-only, bundles + collections). Each node is plotted as 0–100% of its own target so small nodes are directly comparable to large ones. - Change generate_registry_status_reports.py default from auto-commit to --no-commit; --commit must now be passed explicitly to push. - Update CLAUDE.md to reflect the new default. - Regenerate docs/status/ CSVs and HTML with latest data. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Update requirements * Update changelog * Add % completion by node charts and flip commit default to --no-commit - Add "% Completion by Node" section to burnup_chart.html with four normalized line charts (all-versions + latest-only, bundles + collections). Each node is plotted as 0–100% of its own target so small nodes are directly comparable to large ones. - Change generate_registry_status_reports.py default from auto-commit to --no-commit; --commit must now be passed explicitly to push. - Update CLAUDE.md to reflect the new default. - Regenerate docs/status/ CSVs and HTML with latest data. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Update requirements * Update changelog * Update requirements * Update changelog * Remove useless script in docker compose setup (#539) * don't use the local script to launch the API, this is useless and does not match the AWS deployment setup. * trigger local precommit check * migrate precommit configuration * add other missing closing double quote --------- Co-authored-by: Thomas Loubrieu <loubrieu@jpl.nasa.gov> * Update requirements * Update changelog * Add user-facing documentation for registry status reports Explains PDS Keyword Search as the source of truth baseline, the comparison to the Registry API, and adds an FAQ covering node filtering, superseded flag interpretation, and how to request a data release for unreleased bundle/collection versions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Update requirements * Update changelog * Update requirements * Update changelog * Split terraform scripts according to lifecycle of the components (#550) * split terraform scripts according to lifecycle of the components, validated in dev, without terragrunt * split terraform scripts according to lifecycle of the components, validated in dev, without terragrunt --------- Co-authored-by: Thomas Loubrieu <loubrieu@jpl.nasa.gov> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: thomas loubrieu <thomas.loubrieu@jpl.nasa.gov> Co-authored-by: PDSEN CI Bot <pdsen-ci@jpl.nasa.gov> Co-authored-by: al-niessner <1130658+al-niessner@users.noreply.github.com> Co-authored-by: Al Niessner <Al.Niessner@xxx.xxx> Co-authored-by: Jordan Padams <jordan.h.padams@jpl.nasa.gov> Co-authored-by: Jordan Padams <33492486+jordanpadams@users.noreply.github.com> Co-authored-by: Alex Dunn <75815303+alexdunnjpl@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Thomas Loubrieu <loubrieu@jpl.nasa.gov> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: jimmie <jimmie.d.young@gmail.com> Co-authored-by: Rishi Verma <riverma@apache.org> Co-authored-by: edunn <alexander.e.dunn@jpl.nasa.gov>
* Add human-readable Postman collection docs with auto-generation workflow - Add generate_collection_docs.py to convert postman_collection.json to Markdown with TOC, linked TestRail case IDs, and GitHub issue refs - Add generated postman_collection.md alongside the JSON source - Add GitHub Actions workflow to regenerate the doc on every push that touches postman_collection.json (resolves #497) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Exclude postman_collection.md from end-of-file-fixer hook Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Restrict postman docs workflow to feature branches only Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Replace third-party commit action with git CLI commands Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address Copilot review feedback on postman doc generator - Fix TOC anchor mismatch: anchors now include HTTP method to match rendered headings - Add anchor deduplication for repeated request names - Remove unused extract_testrail_ids() function - Add encoding="utf-8" to open() and write_text() for deterministic output - Remove [skip ci] from workflow commit (no loop risk; trigger is .json not .md) - Regenerate postman_collection.md with corrected TOC anchors Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: thomas loubrieu <60993872+tloubrieu-jpl@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
🗒️ Summary
Adds a Python script and GitHub Actions workflow to automatically generate a human-readable Markdown version of
docker/postman/postman_collection.jsonwhenever it is updated in a PR.The generated
postman_collection.mdincludes:Also excludes
postman_collection.mdfrom theend-of-file-fixerpre-commit hook (auto-generated file).🤖 AI Assistance Disclosure
Estimated % of code influenced by AI: 95%
⚙️ Test Data and/or Report
The generated
docker/postman/postman_collection.mdis committed alongside the JSON and can be reviewed directly in this PR.♻️ Related Issues
Fixes #497
🤓 Reviewer Checklist
Documentation and PR Content
Security & Quality
Testing & Validation
Maintenance