Skip to content

Add auto-generated human-readable Postman collection docs - #501

Merged
tloubrieu-jpl merged 6 commits into
developfrom
feature/497-readable-postman
Apr 6, 2026
Merged

Add auto-generated human-readable Postman collection docs#501
tloubrieu-jpl merged 6 commits into
developfrom
feature/497-readable-postman

Conversation

@jordanpadams

Copy link
Copy Markdown
Member

🗒️ Summary

Adds a Python script and GitHub Actions workflow to automatically generate a human-readable Markdown version of docker/postman/postman_collection.json whenever it is updated in a PR.

The generated postman_collection.md includes:

  • Table of contents with folder hierarchy and HTTP method badges
  • Per-request: method, full URL (path variables resolved), Accept header
  • Test assertions with linked TestRail case IDs
  • GitHub issue references as hyperlinks

Also excludes postman_collection.md from the end-of-file-fixer pre-commit hook (auto-generated file).

🤖 AI Assistance Disclosure

  • No AI assistance used
  • AI used for light assistance (e.g., suggestions, refactoring, documentation help, minor edits)
  • AI used for moderate content generation (AI generated some code or logic, but the developer authored or heavily revised the majority)
  • AI generated substantial portions of this code

Estimated % of code influenced by AI: 95%

⚙️ Test Data and/or Report

The generated docker/postman/postman_collection.md is committed alongside the JSON and can be reviewed directly in this PR.

♻️ Related Issues

Fixes #497

🤓 Reviewer Checklist

Documentation and PR Content

  • Documentation: README, Wiki, or inline documentation (Sphinx, Javadoc, Docstrings) have been updated to reflect these changes.
  • Issue Traceability: The PR is linked to a valid GitHub Issue
  • PR Title: The PR title is "user-friendly" clearly identifying what is being fixed or the new feature being added, that if you saw it in the Release Notes for a tool, you would be able to get the gist of what was done.

Security & Quality

  • SonarCloud: Confirmed no new High or Critical security findings.
  • Secrets Detection: Verified that the Secrets Detection scan passed and no sensitive information (keys, tokens, PII) is exposed.
  • Code Quality: Code follows organization style guidelines and best practices for the specific language (e.g., PEP 8, Google Java Style).

Testing & Validation

  • Test Accuracy: Verified that test data is accurate, representative of real-world PDS4 scenarios, and sufficient for the logic being tested.
  • Coverage: Automated tests cover new logic and edge cases.
  • Local Verification: (If applicable) Successfully built and ran the changes in a local or staging environment.

Maintenance

  • Backward Compatibility: Confirmed that these changes do not break existing downstream dependencies or API contracts (or that breaking changes are clearly documented).

jordanpadams and others added 4 commits March 30, 2026 13:46
- Add generate_collection_docs.py to convert postman_collection.json to
  Markdown with TOC, linked TestRail case IDs, and GitHub issue refs
- Add generated postman_collection.md alongside the JSON source
- Add GitHub Actions workflow to regenerate the doc on every push that
  touches postman_collection.json (resolves #497)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@jordanpadams
jordanpadams requested a review from a team as a code owner March 30, 2026 20:47
- Fix TOC anchor mismatch: anchors now include HTTP method to match rendered headings
- Add anchor deduplication for repeated request names
- Remove unused extract_testrail_ids() function
- Add encoding="utf-8" to open() and write_text() for deterministic output
- Remove [skip ci] from workflow commit (no loop risk; trigger is .json not .md)
- Regenerate postman_collection.md with corrected TOC anchors

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds automation to generate and commit a human-readable Markdown rendering of the Postman collection JSON to improve PR reviewability and traceability.

Changes:

  • Added docker/postman/generate_collection_docs.py to convert postman_collection.json into a structured Markdown document (TOC, requests, Accept headers, tests, links).
  • Added a GitHub Actions workflow to regenerate and auto-commit postman_collection.md when the collection JSON changes on non-main branches.
  • Updated pre-commit configuration to exclude the generated Markdown from end-of-file-fixer.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File Description
docker/postman/postman_collection.md New auto-generated Markdown documentation output committed alongside the JSON.
docker/postman/generate_collection_docs.py Script that parses the Postman collection JSON and renders Markdown with TOC, request details, and linkification.
.github/workflows/postman-collection-docs.yml CI workflow to regenerate and commit the Markdown when the collection JSON changes.
.pre-commit-config.yaml Excludes the generated Markdown file from the EOF fixer hook.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +14 to +23
def extract_test_names(test_script: str) -> list[tuple[str, list[str]]]:
"""Return list of (test_name, [testrail_ids]) from pm.test() calls."""
results = []
for match in re.finditer(r'pm\.test\(\s*["\']([^"\']+)["\']', test_script):
name = match.group(1)
ids = re.findall(r"\bC\d{5,}\b", name)
# Strip leading IDs from the display name
display = re.sub(r"^(C\d+\s+)+", "", name).strip()
results.append((display, ids))
return results

Copilot AI Apr 2, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extract_test_names() only detects pm.test() calls where the first argument is wrapped in single/double quotes. The current collection uses template literals (backticks) and ${testrailId} interpolation (e.g., const testrailId = "C4440539"; + pm.test(`${testrailId} …`)), which results in generated docs containing literal ${testrailId} strings and missing TestRail links. Consider expanding parsing to handle backtick strings and, when ${testrailId} is used, extracting the testrailId constant from the same script block so the correct C####### link can be emitted.

Copilot uses AI. Check for mistakes.
Comment on lines +27 to +28
"""Extract GitHub issue refs like NASA-PDS/registry-api#494 from a string."""
return re.findall(r"NASA-PDS/[\w-]+#\d+", name)

Copilot AI Apr 2, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extract_github_refs() only recognizes NASA-PDS/<repo>#<num> patterns. The collection also contains issue references in other common forms (e.g. NASA-PDS/registry-api/issues/66, and NASA-PDS/registry-api/#638), which are currently not converted into hyperlinks in the generated Markdown. Expanding the regex and link rendering to cover these variants would better meet the “GitHub issue references as hyperlinks” requirement.

Suggested change
"""Extract GitHub issue refs like NASA-PDS/registry-api#494 from a string."""
return re.findall(r"NASA-PDS/[\w-]+#\d+", name)
"""Extract GitHub issue refs and normalize them to NASA-PDS/<repo>#<num>."""
pattern = re.compile(
r"(NASA-PDS/([\w-]+)(?:#(\d+)|/#(\d+)|/issues/(\d+)))"
)
refs = []
for match in pattern.finditer(name):
repo = match.group(2)
issue_number = match.group(3) or match.group(4) or match.group(5)
refs.append(f"NASA-PDS/{repo}#{issue_number}")
return refs

Copilot uses AI. Check for mistakes.
Comment on lines +122 to +134
def _build_anchor(heading_text: str, used_anchors: set[str]) -> str:
"""Build a GitHub-style anchor from heading text, ensuring uniqueness."""
anchor = re.sub(r"[^\w\s-]", "", heading_text.lower()).strip()
anchor = re.sub(r"[\s]+", "-", anchor)

if anchor in used_anchors:
suffix = 2
unique_anchor = f"{anchor}-{suffix}"
while unique_anchor in used_anchors:
suffix += 1
unique_anchor = f"{anchor}-{suffix}"
anchor = unique_anchor

Copilot AI Apr 2, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_build_anchor() attempts to mimic GitHub’s duplicate-heading anchor scheme, but it starts suffixing duplicates at -2. GitHub generates #heading, then #heading-1, #heading-2, etc. If duplicate headings ever occur in the collection, the TOC links produced here will not match GitHub’s rendered anchors. Adjust the suffixing to start at 1 for the first duplicate to keep TOC links reliable.

Copilot uses AI. Check for mistakes.
Comment thread .github/workflows/postman-collection-docs.yml Outdated
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

@tloubrieu-jpl tloubrieu-jpl left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jordanpadams , that will help Catherine for the I&T

@tloubrieu-jpl
tloubrieu-jpl merged commit 37d43cb into develop Apr 6, 2026
1 check passed
@tloubrieu-jpl
tloubrieu-jpl deleted the feature/497-readable-postman branch April 6, 2026 17:51
tloubrieu-jpl added a commit that referenced this pull request Jul 22, 2026
* Bump actions/checkout from 5 to 6

Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v5...v6)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump actions/cache from 4 to 5

Bumps [actions/cache](https://github.com/actions/cache) from 4 to 5.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](actions/cache@v4...v5)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump actions/upload-artifact from 5 to 6

Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 5 to 6.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](actions/upload-artifact@v5...v6)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump SonarSource/sonarqube-scan-action from 6 to 7

Bumps [SonarSource/sonarqube-scan-action](https://github.com/sonarsource/sonarqube-scan-action) from 6 to 7.
- [Release notes](https://github.com/sonarsource/sonarqube-scan-action/releases)
- [Commits](SonarSource/sonarqube-scan-action@v6...v7)

---
updated-dependencies:
- dependency-name: SonarSource/sonarqube-scan-action
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix end of file

* Update requirements

* Update changelog

* Commiting /github/workspace/src/pds/registry/VERSION.txt for stable release

* Setting next dev version to 1.7.0

* switch to dev version of components

* make integration tests work again

* downgrade sweeper because latest stables builds are broken and do not work with the latest version of api

* Update postman_collection.json (#470)

* update registry testing

First, modified .env and docker-compose to allow the user to move from a docker volume to a disk with more space.

Second, added 8 tests for "exists" capability in the registry-api query language

* Update postman_collection.json

Change the way wildcarding is expressed.

* Update postman_collection.json

Put * back in because of too happy to delete.

---------

Co-authored-by: Al Niessner <Al.Niessner@xxx.xxx>
Co-authored-by: thomas loubrieu <thomas.loubrieu@jpl.nasa.gov>

* fix wrong automated test

* Update missing products reports - 2026-01-28 10:08:52

* Update README.md

* Create requirements.txt

* Update README.md

* Update Integration test to account for the refactoring of the ancestry management (#474)

* update properties count test

* disable 2nd-order membership tests as this functionality is deprecated and must be reimplemented or removed at a later date

* update properties count test again- this does not appear to be consistent and may be picking up metadata keys

* use latest verion of sweeper after successful integration tests

* Add AOSS vs. Managed OpenSearch Trade Study

Refs #451

* Fix bug in requirements.txt

* Update missing products reports - 2026-03-03 15:55:00

* Update missing products reports - 2026-03-04 13:55:14

* Update queries to include up to 10000 records

Resolves #471

* Update missing products reports - 2026-03-04 13:57:11

* Bump actions/upload-artifact from 6 to 7

Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 6 to 7.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](actions/upload-artifact@v6...v7)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update missing products reports - 2026-03-18 12:06:51

* Update missing products reports - 2026-03-18 12:07:22

* Update missing products reports - 2026-03-18 12:13:30

* Update missing products reports - 2026-03-18 13:06:49

* Split missing products reports by latest/superseded version and add burndown tracking

- generate_registry_status_reports.py: for missing bundles/collections, produce
  three CSVs per type (overall, *_latest_*, *_superseded_*) by grouping LIDVIDs
  by LID and comparing versions numerically so 3.9 < 3.13. Resolves
  pds-registry-client via sys.executable rather than shell PATH to avoid venv
  activation issues. Appends one row per run to docs/status/counts_history.csv
  for burndown chart tracking (append-only, never overwritten).
- backfill_history.py: new one-off script to populate counts_history.csv from
  git history of the status CSVs; idempotent (skips dates already present).
- docs/status/counts_history.csv: initial 5-snapshot backfill from git history
  (2025-11-21 through 2026-03-18).
- docs/status/README.md: updated metrics table to show Latest/Superseded/Total
  columns; added Historical Counts section documenting the burndown CSV format.
- .gitignore: ignore *.code-workspace files.
- CLAUDE.md: document the status reporting scripts and operational notes.

Closes #481. Relates to #476.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Add API validator script to spot-check missing products

validate_missing_in_api.py reads the overall missing bundle and collection
CSVs, randomly samples up to 500 LIDVIDs (configurable via --sample/--seed),
queries https://pds.nasa.gov/api/search/1/products/{lidvid} for each, and
reports HTTP status codes. Products returning 200 are flagged as unexpectedly
present in the API.

Features:
- Concurrent requests via ThreadPoolExecutor (--workers, default 10)
- Retry logic for transient errors (429, 5xx)
- Results written to docs/status/validation_results.csv
- --dry-run, --found-only, --sample 0 (full pool) modes
- Fixed seed support for reproducible sampling

Closes #484. Relates to #476.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Update scripts/generate_registry_status_reports.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Initial plan

* Initial plan

* Initial plan

* Tighten type hints in split_by_version() in backfill_history.py

Co-authored-by: jordanpadams <33492486+jordanpadams@users.noreply.github.com>
Agent-Logs-Url: https://github.com/NASA-PDS/registry/sessions/ab42f1c3-5d84-40a5-b8c0-c00a2938c827

* Initial plan

* Use --detach instead of -d in docker compose command in CLAUDE.md

Co-authored-by: jordanpadams <33492486+jordanpadams@users.noreply.github.com>
Agent-Logs-Url: https://github.com/NASA-PDS/registry/sessions/3ad3b701-4ef9-4645-af27-3b698844b1e7

* Compute CSV counts once and reuse in both generate_metrics_from_csvs and append_history_row

Co-authored-by: jordanpadams <33492486+jordanpadams@users.noreply.github.com>
Agent-Logs-Url: https://github.com/NASA-PDS/registry/sessions/00d36611-6202-4258-9b9c-e5fa636f42da

* Fix --volume to --volumes in CLAUDE.md docker compose command

Co-authored-by: jordanpadams <33492486+jordanpadams@users.noreply.github.com>
Agent-Logs-Url: https://github.com/NASA-PDS/registry/sessions/42ce8b7b-f5ac-4e73-b3a6-b5f5fe6d9dc4

* Consolidate missing products CSVs into single file with superseded column

Instead of generating three separate CSVs per product type (overall, latest,
superseded), generate one CSV with a `superseded` column (true/false) indicating
whether each LIDVID is the latest version for its LID.  History counts and README
metrics table are unchanged.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Update missing products reports - 2026-03-23 18:57:33

* Remove CSVs we are no longer generating

* Update missing products reports - 2026-03-24 11:40:39

* Update missing products reports - 2026-03-24 12:06:47

* Update missing products reports - 2026-03-24 12:45:49

* Update missing products reports - 2026-03-24 13:54:53

* Update missing products reports - 2026-03-24 14:41:16

* Fix superseded version detection for missing bundles and collections

Queries now fetch all products (not just found_in_registry=false) so that
version ordering is determined across the full set. Python filters to missing
rows after annotating superseded status, ensuring a missing LIDVID is correctly
marked superseded when a higher version exists in the registry.

Also adds CSV headers to output files and skips header rows in count aggregation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* skip broken tests

* skip broken tests

* down to a minor mismatch

* minimal skips

* Improve registry documentation for new users (#392, #391, #393, #394) (#502)

This commit addresses multiple documentation issues to better support new PDS users:

**Issue #391 - Add links to component documentation:**
- Added link to registry-client external docs from Registry Client section
- Added links from Registry Manager to update_status, delete_data, and create_reg pages
- Added link from Harvest section to load1.html user guide

**Issue #393 - Improve documentation consistency and ordering:**
- Reordered components: Harvest → Registry Manager → Registry Client → OpenSearch → Registry API
- Made Authentication/Authorization a subsection of OpenSearch
- Renamed "API" to "Registry API (PDS Search API)" for clarity and consistency
- Added note about archive status affecting API visibility
- Clarified archive status values with descriptions:
  * archived - publicly visible
  * certified - publicly visible
  * restricted - not publicly visible
  * staged - default, not publicly available
- Fixed inconsistent terminology: changed "uploaded" to "loaded" in load1.rst

**Issue #394 - Update registry-manager documentation:**
- Updated parameter documentation to reflect current CLI (2 required, 2 optional)
- Replaced deprecated -es with -registry parameter
- Updated examples to use new parameter names
- Fixed "registered" to "certified" in Registry API section

These changes improve discoverability and clarity for users new to the PDS Registry system.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* Add auto-generated human-readable Postman collection docs (#501)

* Add human-readable Postman collection docs with auto-generation workflow

- Add generate_collection_docs.py to convert postman_collection.json to
  Markdown with TOC, linked TestRail case IDs, and GitHub issue refs
- Add generated postman_collection.md alongside the JSON source
- Add GitHub Actions workflow to regenerate the doc on every push that
  touches postman_collection.json (resolves #497)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Exclude postman_collection.md from end-of-file-fixer hook

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Restrict postman docs workflow to feature branches only

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Replace third-party commit action with git CLI commands

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: address Copilot review feedback on postman doc generator

- Fix TOC anchor mismatch: anchors now include HTTP method to match rendered headings
- Add anchor deduplication for repeated request names
- Remove unused extract_testrail_ids() function
- Add encoding="utf-8" to open() and write_text() for deterministic output
- Remove [skip ci] from workflow commit (no loop risk; trigger is .json not .md)
- Regenerate postman_collection.md with corrected TOC anchors

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: thomas loubrieu <60993872+tloubrieu-jpl@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* update exists operator tests as NASA-PDS/registry-api#741

* update test for updated members/members and member-of/member-of support

* add missing host for elasticsearch service for docker compose to work with option --project-name

* simplify the api launch

* fix docker compose for github action support

* use --project-name value in the docker compose through a new env variable

* add missing numbering suffix

* add healthcheck on api and use it as a dependency for other services to start.

* simplify docker compose, adjust test or latest PR of registry-api #753

* change test expectation for GONE members/members end-point

* add check for good error message

* docs: regenerate postman_collection.md

* non existance has changed

* docs: regenerate postman_collection.md

* fixup last test failure

* docs: regenerate postman_collection.md

* Update missing products reports - 2026-04-27 14:15:27

* Bump SonarSource/sonarqube-scan-action from 7 to 8

Bumps [SonarSource/sonarqube-scan-action](https://github.com/sonarsource/sonarqube-scan-action) from 7 to 8.
- [Release notes](https://github.com/sonarsource/sonarqube-scan-action/releases)
- [Commits](SonarSource/sonarqube-scan-action@v7...v8)

---
updated-dependencies:
- dependency-name: SonarSource/sonarqube-scan-action
  dependency-version: '8'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Initial import of terraform for provisioned opensearch

* Correct committed versions of main and variable tf files.

* Initial commit of index template tf scripts.

* Fix opensearch provider.

* Add pipeline terraform.

* Make main provider.tf consistent.

* Make index_templates provider.tf consistent.

* Intermediate commit (to preserve content) of OSI pipeline tf scripts.

* Switched strategy for osi pipeline config body - went to an external yaml file.

* Fixed specification of policy file variable.

* Update missing products reports - 2026-05-11 13:35:12

* Revise pipeline terraform scripts.

* Add burnup charts, loaded product reports, and per-node tracking

- Add burnup_chart.html with interactive time-series visualization
- Add burnup_history.csv, burnup_by_node.csv and their *_latest variants
- Add loaded_bundles/collections CSVs and per-node OpenSearch DSL queries
- Update generate_registry_status_reports.py to produce all new reports
- Update docs/status/README.md to document new outputs

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

* Add workflow to publish burnup chart to GitHub Pages

- New publish-burnup-chart.yml triggers on push to main when
  docs/status/burnup_chart.html changes, copying it to gh-pages/status/
- Add docs/source/status.rst with link to the live chart
- Wire status.rst into the Sphinx toctree in index.rst

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

* Add burnup chart publishing to GitHub Pages

- Fix publish-burnup-chart workflow: save burnup_chart.html to /tmp before
  checking out gh-pages branch so the file survives the workspace replacement
- Add docs/source/status.rst linking to the live chart
- Wire status.rst into the Sphinx toctree
- Add docs/status/burnup_chart.html to main so the workflow can trigger

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

* Update publish-burnup-chart.yml

* Update CLAUDE.md and docs/status/README with loaded product and burnup chart documentation

- Expand Registry Status Reporting in CLAUDE.md: correct venv invocation,
  table of all six query config files with endpoints, data population
  distinction (legacy Solr vs new OpenSearch / PSA inflation), AOSS
  pagination caveat, full output file table, burnup chart details
- Add "How Numbers Are Calculated" section to docs/status/README.md
  explaining missing/loaded/staged/burnup derivation and the superseded
  flag algorithm
- Add Loaded Products and Burnup Charts report sections to README
- Update download script with new loaded_* and burnup_* files

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

* Update metrics

* Fix publish workflow to checkout triggering branch instead of hardcoded main

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

* Improve user documentation with procedural step-by-step instructions

Rewrites connection-setup, installation, and all user task docs to use
numbered steps throughout. Adds test_connection page, standardizes config
file naming to registry-{tool}-config-{node}-{venue} pattern under ~/.pds/,
and fixes autosectionlabel warnings in conf.py.

Resolves #515

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

* Switch over to full block for index template settings to provide more flexibility

* Fix typo for pipeline role arn.

* Add index alias support.

* Update missing products reports - 2026-06-09 11:21:52

* Expand registry loading status page with burnup chart and report links

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Scale x-axis tick density to selected date range on burnup chart

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Fix burnup chart x-axis to use calendar time scale independent of data

Switch from category scale to Chart.js time scale with chartjs-adapter-date-fns.
The axis now spans the full selected date range with evenly-spaced calendar
ticks (weekly for ≤6mo, monthly for ≤2yr, yearly for all time) regardless
of where data points happen to fall.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Update missing products reports - 2026-06-11 08:05:34

* Update changelog

* Update reports

* Update changelog

* add test for security patch, update docker compose for latest sweeper docker upgrade

* docs: regenerate postman_collection.md

* update test after sweeper upgrade

* add explicit pull_policy for registry-api to make sure the local image is used when available

* Fix race condition: add depends_on between domain policy and domain

Without this, Terraform may attempt to create the domain access policy
before the OpenSearch domain exists, causing a ResourceNotFoundException.

* Bump actions/checkout from 6 to 7

Bumps [actions/checkout](https://github.com/actions/checkout) from 6 to 7.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v6...v7)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump actions/cache from 5 to 6

Bumps [actions/cache](https://github.com/actions/cache) from 5 to 6.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](actions/cache@v5...v6)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add script to detect and delete stale LDD_Info sentinel records

Adds scripts/detect_stale_ldd_sentinels.py which queries all node
registry -dd indexes for LDD_Info sentinels that have zero field
documents, reports them with ready-to-paste Dev Tools delete queries,
and optionally deletes them with explicit confirmation.

Key features:
- Paginates sentinel fetch with search_after so all records are found
- Checkpoints progress after each _count query; interrupted runs resume
- Retries transient connection errors with exponential backoff
- --delete flag required for any writes; prompts for 'yes' confirmation
- --report / --reset / --checkpoint flags for operational control

Fixes: NASA-PDS/registry-loader#89

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Re-enable secrets detection using standard detect-secrets

Replaces the disabled slim-detect-secrets GitHub Action and commented-out
pre-commit hook with the updated approach from cloud-tools:
- New `scripts/detect_secrets_baseline.sh` with scan/audit/check modes,
  `.detect-secrets-ignore` support, and Python-based comparison (no jq)
- Workflow delegates to the script instead of inlining logic
- `.detect-secrets-ignore` excludes docker/default-config, docs, test data
- `.secrets.baseline` regenerated with detect-secrets~=1.5.0; all 3 findings
  audited as false positives (Jenkins credential ref, test placeholder, md5 URL param)
- Pre-commit hook re-enabled pointing to the new script

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Fix --reset flag skipping sentinel fetch

load_checkpoint returned {"sentinels": [], "results": {}} as the default
when no file exists, so after --reset deleted the checkpoint the reload
saw "sentinels" was present and skipped the fetch entirely, printing
"Resuming from checkpoint — 0 sentinel(s) already fetched" / "Nothing to do."

Fix: return {"results": {}} so a fresh/reset checkpoint has no "sentinels"
key and the fetch is always triggered.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Fix status report showing false 'not yet checked' warning

print_status_report compared len(results) against len(sentinels) (raw hit
count), but results deduplicates by (index, namespace) key. With 420 raw
hits but only 164 unique pairs, the report falsely reported 256 unchecked.

Fix: compute total from unique (index, ns) keys in the sentinels list so
the denominator matches what results actually tracks.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Update secrets and dependencies

* Fix linter issues

* Update .gitignore

* Add terraform script for complete deployment of the Registry in the Dev Account (#475)

* initialize terraform to deploy simple regitry in dev, wip

* wip: terraform deployment of opensearch serverless collection

* make the depoyment work with external definition of IAM role in pds-cds-infra repository

* adjust the role/policy management without using SSM for now

* upgrade provider for compatibility purpose

* modularize terraform, to make external policies stand out

* add missing module directory

* clean role argument names

* rename opensearch api and dashboard access policy

* wip: untested credentials endpoint for registry-tools in terraform

* automate registry initialization with registry-mgr/harvest

* add registry-manager set-archive-status command to match reference test setup

* add usage documentation in readme.

* Update missing products reports - 2026-03-18 10:01:42

* Update missing products reports - 2026-03-18 12:04:37

* add missing api gateway and first version of automated initialization with test data on AWS

* add creation of the build directory

* validate terraform apply/destroy

* add build layer directory for plan phase.

* use pip3 instead of pip which might not be available locally.

* wip: registry deployed in dev with api and sweeper manually added from their own terraform, integration test still fail.

* remove automated lias creation as it does not work yet

* add comment for commented out code, specifically dedicated to Sean!

* update to initialize node registry without ref data.

* update to have log when the lambda layer is build as it does not work from terragrunt call

* try to clarify what path module is

* force to recreate the layer whenever the zip file has to be redeployed

* minor fixes, logs

* add sleep to make sure the harvest test products are properly indexed

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Fix COGNITO_ALLOWED_GROUPS substring matching by splitting into a set

* Strip whitespace from COGNITO_ALLOWED_GROUPS entries after splitting

* force delete registry indexes before deployment, make vpc endpoint service name unique to handle multiple deployment in the same account

* update sweeper call arguments per later update on sweeper docker image

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* update docker compose to make the test wait for api to be ready

* rename opensearch module as opensearch_serverless module to avoid conflicts

* make it compliant with python 3.12

* rename opensearch directory into opensearch_managed for consistency with opensearch_serverless

* fix renamed vpc endpoint service

* add the vpc endpoint with name depending on the opensearch collection name

* finally remove the collection service because it cannot work

* add missing relay variable

* Linter updates

* fix obsolete README instruction

---------

Co-authored-by: thomas loubrieu <thomas.loubrieu@jpl.nasa.gov>
Co-authored-by: Thomas Loubrieu <loubrieu@jpl.nasa.gov>
Co-authored-by: Jordan Padams <33492486+jordanpadams@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Jordan Padams <jordan.h.padams@jpl.nasa.gov>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

* Bump python-jose from 3.3.0 to 3.4.0 in /terraform/lambda/src/layer

Bumps [python-jose](https://github.com/mpdavis/python-jose) from 3.3.0 to 3.4.0.
- [Release notes](https://github.com/mpdavis/python-jose/releases)
- [Changelog](https://github.com/mpdavis/python-jose/blob/master/CHANGELOG.md)
- [Commits](mpdavis/python-jose@3.3.0...3.4.0)

---
updated-dependencies:
- dependency-name: python-jose
  dependency-version: 3.4.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump requests from 2.32.3 to 2.33.0 in /terraform/lambda/src/layer

Bumps [requests](https://github.com/psf/requests) from 2.32.3 to 2.33.0.
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](psf/requests@v2.32.3...v2.33.0)

---
updated-dependencies:
- dependency-name: requests
  dependency-version: 2.33.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update dependabot.yml

* Update dependabot.yml

* Update unstable-cicd.yaml

* Update branch-cicd.yaml

* Update missing products reports - 2026-07-13 13:56:58

* Fix linter issues

* Bump actions/checkout in the github-actions group across 1 directory

Bumps the github-actions group with 1 update in the / directory: [actions/checkout](https://github.com/actions/checkout).


Updates `actions/checkout` from 4 to 7
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v4...v7)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update requirements

* Update changelog

* Update CLAUDE.md to require pre-commit install or tox before pushing

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Update requirements

* Update changelog

* update sweepers command to reflect ENTRYPOINT directive introduced in 2b83e392

* Update requirements

* Update changelog

* Add % completion by node charts and flip commit default to --no-commit

- Add "% Completion by Node" section to burnup_chart.html with four
  normalized line charts (all-versions + latest-only, bundles +
  collections). Each node is plotted as 0–100% of its own target so
  small nodes are directly comparable to large ones.
- Change generate_registry_status_reports.py default from auto-commit
  to --no-commit; --commit must now be passed explicitly to push.
- Update CLAUDE.md to reflect the new default.
- Regenerate docs/status/ CSVs and HTML with latest data.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Update requirements

* Update changelog

* Add % completion by node charts and flip commit default to --no-commit

- Add "% Completion by Node" section to burnup_chart.html with four
  normalized line charts (all-versions + latest-only, bundles +
  collections). Each node is plotted as 0–100% of its own target so
  small nodes are directly comparable to large ones.
- Change generate_registry_status_reports.py default from auto-commit
  to --no-commit; --commit must now be passed explicitly to push.
- Update CLAUDE.md to reflect the new default.
- Regenerate docs/status/ CSVs and HTML with latest data.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Update requirements

* Update changelog

* Update requirements

* Update changelog

* Remove useless script in docker compose setup (#539)

* don't use the local script to launch the API, this is useless and does not match the AWS deployment setup.

* trigger local precommit check

* migrate precommit configuration

* add other missing closing double quote

---------

Co-authored-by: Thomas Loubrieu <loubrieu@jpl.nasa.gov>

* Update requirements

* Update changelog

* Add user-facing documentation for registry status reports

Explains PDS Keyword Search as the source of truth baseline, the
comparison to the Registry API, and adds an FAQ covering node filtering,
superseded flag interpretation, and how to request a data release for
unreleased bundle/collection versions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Update requirements

* Update changelog

* Update requirements

* Update changelog

* Split terraform scripts according to lifecycle of the components (#550)

* split terraform scripts according to lifecycle of the components, validated in dev, without terragrunt

* split terraform scripts according to lifecycle of the components, validated in dev, without terragrunt

---------

Co-authored-by: Thomas Loubrieu <loubrieu@jpl.nasa.gov>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: thomas loubrieu <thomas.loubrieu@jpl.nasa.gov>
Co-authored-by: PDSEN CI Bot <pdsen-ci@jpl.nasa.gov>
Co-authored-by: al-niessner <1130658+al-niessner@users.noreply.github.com>
Co-authored-by: Al Niessner <Al.Niessner@xxx.xxx>
Co-authored-by: Jordan Padams <jordan.h.padams@jpl.nasa.gov>
Co-authored-by: Jordan Padams <33492486+jordanpadams@users.noreply.github.com>
Co-authored-by: Alex Dunn <75815303+alexdunnjpl@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Thomas Loubrieu <loubrieu@jpl.nasa.gov>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: jimmie <jimmie.d.young@gmail.com>
Co-authored-by: Rishi Verma <riverma@apache.org>
Co-authored-by: edunn <alexander.e.dunn@jpl.nasa.gov>
tloubrieu-jpl added a commit that referenced this pull request Jul 28, 2026
* Add human-readable Postman collection docs with auto-generation workflow

- Add generate_collection_docs.py to convert postman_collection.json to
  Markdown with TOC, linked TestRail case IDs, and GitHub issue refs
- Add generated postman_collection.md alongside the JSON source
- Add GitHub Actions workflow to regenerate the doc on every push that
  touches postman_collection.json (resolves #497)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Exclude postman_collection.md from end-of-file-fixer hook

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Restrict postman docs workflow to feature branches only

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Replace third-party commit action with git CLI commands

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: address Copilot review feedback on postman doc generator

- Fix TOC anchor mismatch: anchors now include HTTP method to match rendered headings
- Add anchor deduplication for repeated request names
- Remove unused extract_testrail_ids() function
- Add encoding="utf-8" to open() and write_text() for deterministic output
- Remove [skip ci] from workflow commit (no loop risk; trigger is .json not .md)
- Regenerate postman_collection.md with corrected TOC anchors

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: thomas loubrieu <60993872+tloubrieu-jpl@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

As a developer, I want human-readable Postman collection docs auto-generated on update

3 participants