Port registry-common#296 and #303: warn on unresolvable LDD association and fix AOSS delete-by-query pagination - #123
Merged
Conversation
…ociation, fix AOSS DBQ pagination - ClassAttrAssociationParser: replace throw with warn+skip when an association has isAttribute=true but no resolvable attribute ID (neither 'attributeId' nor 'identifier' key found). Adds Logger; removes now-unused LddException import. Ports NASA-PDS/registry-common#296 (refs #88). - RestClientWrapper._performDBQRequest(): loop until the search page is empty rather than doing a single pass. AOSS eventual consistency means deleted docs remain visible to subsequent searches, so a single-pass search-then-delete silently leaves documents behind. Adds INFO-level progress logging per batch; per-doc detail at DEBUG. Ports NASA-PDS/registry-common#303. Note: registry-common#295 (JsonLddLoader sentinel guard) was already applied to this repo in the issues/81-aoss-ldd-propagation-race merge. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…over newer version - DataLoader: add ignoreConflicts flag (default false). When true, 409 Conflict on "create" is logged at DEBUG and treated as success rather than counted as a failure. Also improves the error message to name the index and count of truly failed documents. - JsonLddLoader: enable ignoreConflicts=true on the -dd loader so that loading an older LDD version over an already-loaded newer version no longer produces a misleading "Failed to upload all documents" ERROR. Adds INFO log when the requested LDD is older than what is already in the registry. - SchemaUpdater: demote log.error to log.warn in the updateSchema catch block (harvest continues — not a product failure). In updateLdd catch blocks, move ERROR log inside the lddInfo.isEmpty() guard so it only fires when a product will actually fail; the has-fallback branch uses WARN. Convert format strings to parameterised log4j2 style throughout. Ports NASA-PDS/registry-common#304 (fixes NASA-PDS/harvest#342). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…dVersions sentinel, add tests - LddVersions: promote DEFAULT_DATE to public; add public DEFAULT_LAST_DATE constant (Instant) so callers don't repeat the sentinel string literal. - JsonLddLoader: replace hardcoded "1965-01-01..." with LddVersions.DEFAULT_LAST_DATE; add parse-check guard so the "not newer" log is only emitted when the LDD date is actually parseable (avoids misleading output on bad date strings); demote from INFO to DEBUG. - DataLoader: correct the comment on the uploaded!=numRecords branch — clarifies that when ignoreConflicts=false, 409s ARE counted as failures (a shortfall means real write failures). Pass ex as trailing arg to log.warn in SchemaUpdater so the full stack trace is recorded. - SchemaUpdater: pass exception object as trailing log arg so stack trace is recorded. - TestDataLoaderIgnoreConflicts: new unit test covering LddVersions sentinel constant consistency, 409 counted/not-counted per ignoreConflicts flag, non-409 errors always counted, and successful items producing zero errors. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ion suite New test class covering the harvest#342 scenario end-to-end without a live cluster: - lddDateToIsoInstant: confirms no-timezone dates (e.g. "2022-09-19T07:35:36") parse correctly and compare as older than timezone-qualified dates. - createEsDataFile with older LDD: asserts "create" action lines in NDJSON — these yield 409s on pre-existing docs, accepted silently by ignoreConflicts=true. - createEsDataFile with newer LDD: asserts "index" action lines so existing field definitions are updated. - createEsDataFile with no prior LDD (epoch sentinel): asserts "index" lines for a first-ever load. - No-timezone date in LDD treated as older than newer registry date: the exact regression from harvest#342. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
al-niessner
approved these changes
Jul 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🗒️ Summary
Ports three fixes from
registry-commoninto the monorepo'scommonmodule:registry-common#296 →
ClassAttrAssociationParser:Replaces the
throwwith a warn-and-skip when an association hasisAttribute=truebut no attribute ID can be resolved (neither"attributeId"nor"identifier"key present). Previously the parser threwLddException, halting the entire LDD load. Now it logs a warning and continues, matching the registry-common behaviour and allowingJsonLddLoader's zero-field sentinel guard (#295, already present here) to work correctly on the next run. Also corrects the comment: the association key format is determined by LDD tooling version, not IM version.registry-common#303 →
RestClientWrapper._performDBQRequest():Loops until the search page comes back empty rather than doing a single pass. AOSS eventual consistency means deleted docs remain visible to subsequent searches, so a single-pass search-then-delete silently leaves documents behind on every call. Adds INFO-level progress logging (one line per batch) and DEBUG-level per-doc logging.
registry-common#295 (
JsonLddLoadersentinel guard): Already present in this repo (merged via issues/81-aoss-ldd-propagation-race). No changes needed.🤖 AI Assistance Disclosure
Estimated % of code influenced by AI: 70%
⚙️ Test Data and/or Report
ClassAttrAssociationParserchange: covered by existingTestLddFieldResolutionparameterised tests (MRO 1M00_1400 and CART 1Q00_1970 fixtures). The warn path is exercised when an association hasisAttribute=truebut no resolvable ID.RestClientWrapperchange: validate by runningregistry-manager delete-dd -ns <namespace>against an AOSS node with multiple pages of docs; confirm all are deleted (count reaches 0) and no change in behaviour against standard OpenSearch.♻️ Related Issues
Refs #88 (LDD fields silently not loaded — ClassAttrAssociationParser fix)
Refs NASA-PDS/registry-common#296
Refs NASA-PDS/registry-common#303
🤓 Reviewer Checklist
Reviewers: Please verify the following before approving this pull request.
Documentation and PR Content
Security & Quality
Testing & Validation
Maintenance