Conversation
Covers the round trip for tags/glossary terms with special characters, column tag removal, orphan default-graph triples, and bulk add/remove to assets for both tags and glossary terms. Confirmed live against unmodified main: the bulk add/remove and column-tag-removal cases fail today because RdfTagUpdater's inline SPARQL never syncs those paths. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
RdfTagUpdater hand-built SPARQL with String.format to write/remove om:hasTag / om:hasGlossaryTerm inline on every tag mutation. It built invalid IRIs for tag FQNs with special characters (the crash behind Sentry 7735060699), used non-canonical URIs even when it parsed, and wrote to Fuseki's default graph instead of the named knowledge graph. None of it was necessary: EntityRepository.postUpdate/postCreate already call RdfUpdater.updateEntity, which reloads the entity and reconciles its tag triples with canonical URIs in the right graph. Deleted the inline writer and its SPARQL-building helpers from RdfTagUpdater, keeping only the deferral scope the create-flush collector still opens/drains (now a no-op; left in place rather than torn out, see follow-up note). Dropped the corresponding RDF branches from EntityRepository's tag-apply methods, collapsed the targetType/ targetId overloads that existed only to feed the old writer, and renamed applyTags*InFlushAndDeferRdf / applyTagsBatchWithRdf to drop the now-inaccurate suffixes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
With RdfTagUpdater's inline writer gone, RdfUpdater.updateEntity (the async snapshot writer) is now the only thing that writes tag triples. Most tag-mutation paths already trigger it via postUpdate/postCreate, but a few never did — they only wrote to the DB and search index, so RDF silently went stale: - TagRepository/GlossaryTermRepository bulk add/remove-to-assets (including the column and glossary-term-self-tag cases): the remove side had no RDF sync at all, even before this change. - TaskWorkflowHandler's three tag-write fallbacks (merge/tier/ suggestion) that skip the normal patch path. - TableRepository.addDataModel, which bypasses EntityRepository.update() entirely (documented in its own comment) and so never ran postUpdate. Not in the original call-site audit; found because it used to rely on RdfTagUpdater and would have silently lost RDF sync once that writer was removed. Added RdfUpdater.updateEntity(...) next to each existing searchRepository.updateEntity(...) call so the two indices stay in sync going forward. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found via the columnTagRemoval_clearsCanonicalTriple IT while
verifying the RdfTagUpdater removal: JenaFusekiStorage's predicate-
scoped DELETE (buildPredicateScopedDelete) only scopes to the entity's
own URI plus whatever RdfOwnedResources.ownedTriplesPattern() matches,
which covered hasCustomProperty and hasExtension/hasExtensionProperty
owned subresources but never column URIs. emitColumns links
tableResource om:hasColumn columnResource and writes fresh om:hasTag
triples onto each column resource, so removing a column's tag never
deleted the stale triple on that column's canonical URI — it was
orphaned permanently, independent of RdfTagUpdater's bugs. Reproduced
live via both a table-nested PATCH and the dedicated
/v1/columns/name/{fqn} endpoint, ruling out a client-diffing artifact.
Added a third UNION branch to ownedTriplesPattern() mirroring the
existing two, walking om:hasColumn then zero-or-more om:hasChildColumn
hops so nested/struct columns are covered too.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
✅ PR checks passedThe linked issue has a description and all required Shipping project fields set. Thanks! |
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
Per PR review: awaitAsk/awaitAskFalse caught the broad Exception type instead of the specific one, violating java.md's error-handling rule. Awaitility.await().until(...) only ever throws ConditionTimeoutException here (the polled condition itself never throws — RdfTestUtils.executeSparqlAsk swallows its own exceptions and returns false), so narrow the catch to that. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
✅ Playwright Results — workflow succeededValidated commit ✅ 4490 passed · ❌ 0 failed · 🟡 6 flaky · ⏭️ 1 skipped · 🧰 0 lifecycle flaky PerformanceBlocking targets: ✅ met · Optimization targets: 🟡 in progress Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting. 🕒 Full workflow signal wall (to summary) 59m 48s ⏱️ Max setup 4m 16s · max shard execution 21m 40s · max shard-job elapsed before upload 24m 59s · reporting 19s 🌐 221.45 requests/attempt · 2.23 app boots/UI scenario · 42.51% common-shard skew Optimization targets still in progress:
🟡 6 flaky test(s) (passed on retry)
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
Code Review ✅ Approved🟡 Medium risk Removes the invalid inline SPARQL tag writer from OptionsDisplay: compact → Counting what did not apply, without listing it. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source |
Describe your changes:
Fixes #33474
Removing a tag or glossary term from an entity with RDF enabled logged
org.apache.jena.query.QueryParseExceptionfromRdfTagUpdater.removeTagInline(Sentry: https://collate-3b.sentry.io/issues/7735060699/). The PATCH still succeeded because the exception was caught and logged, but the RDF sync was wrong in more ways than the parse error:RdfTagUpdaterbuilt SPARQL withString.format, concatenating the tag FQN into an IRI unescaped. A tag name with a space (or any char the SPARQLIRIREFgrammar forbids) produced an invalid IRI and crashed the parser.tag/<fqn-as-path>and a hash-basedentity/<guessed-type>/<fqn.hashCode()>, never the canonicalentity/tag/{uuid}/entity/{type}/{uuid}URIsRdfPropertyMapperactually reads/writes.graph/knowledgegraph. Withtdb2:unionDefaultGraph true, these orphan triples leaked into every query that doesn't scope toGRAPH ?g.JenaFusekiStorage's reconciliation DELETE (RdfOwnedResources.ownedTriplesPattern()) never wiped stale triples on column URIs, so removing a column's tag left it orphaned in RDF forever — independent ofRdfTagUpdater.None of
RdfTagUpdater's inline writing was necessary:EntityRepository.postUpdate/postCreatealready callRdfUpdater.updateEntity, the async snapshot writer that reloads the entity and reconciles its tag triples with canonical URIs in the right graph, for the entity and its columns. This PR removes the inline writer entirely and makes the snapshot writer the single source of truth for tag triples, then closes the few paths that had no snapshot write after them.Type of change:
High-level design:
RdfTagUpdater: deletedapplyTag/removeTag,applyTagInline/removeTagInline, and all their SPARQL-building helpers (fqnToUri,toTargetUri,resolveTagInfo,resolveGlossaryTermId,inferEntityType,escapeSparqlString,TagInfo). Kept the deferral scope (beginDeferral/checkpoint/drainDeferredToList/runDeferredClosures/clearDeferred) in place —EntityRepository's create-flush collector still opens/drains it, and it's now a harmless no-op (always drains an empty list). Left as a documented follow-up rather than torn out here, to keep this diff reviewable.EntityRepository: removed the RDF branches fromapplyTags/applyTagsAdd/applyTagsDelete/applyTagsBatch(renamed fromapplyTagsBatchWithRdf) and the innerEntityUpdater'sapplyTags*InFlushAndDeferRdfmethods (renamed toapplyTags*InFlush, dropping the now-inaccurate suffix). Collapsed thetargetType/targetIdoverloads, which existed only to feed the old writer and had no other callers once it's gone.TagRepository/GlossaryTermRepository: addedRdfUpdater.updateEntity(...)next to every existingsearchRepository.updateEntity(...)in the bulk add/remove-to-assets paths (entity, column, and glossary-term-self-tag cases) — these never triggered a snapshot write at all, so RDF silently went stale on every bulk operation, most notably every bulk remove.TaskWorkflowHandler: added the same call after its three tag-write fallbacks (merge/tier/suggestion) that bypass the normal patch path.TableRepository.addDataModel: also bypassesEntityRepository.update()entirely (documented in its own existing comment) and never ranpostUpdate. Not in the original call-site list — found because it used to rely onRdfTagUpdaterand would have silently lost RDF sync once that writer was removed.RdfOwnedResources: added a thirdownedTriplesPattern()branch walkingom:hasColumnthen zero-or-moreom:hasChildColumnhops (for nested/struct columns), mirroring the existinghasCustomProperty/hasExtensionbranches, so the reconciliation DELETE actually wipes stale column-level triples.Alternative considered and rejected: fixing
RdfTagUpdater's SPARQL in place (proper escaping,GRAPHclause, canonical URIs shared withRdfPropertyMapper). Rejected because that leaves two independent writers for the same predicates racing the async snapshot writer, with no ordering guarantee — worse to maintain than removing one of them.Existing orphan data: deployments that already hit this bug have orphan default-graph triples. Not shipping a cleanup script in this PR — an RDF rebuild (
rdf/rebuild/RdfDatasetManager) already produces a clean dataset from scratch, which is the documented remediation for already-affected deployments.Collate: grepped
openmetadata-collate(outside the nestedOpenMetadata/ai-platformsubmodules) forRdfTagUpdaterand every renamed method (applyTagsAddInFlushAndDeferRdf,applyTagsDeleteInFlushAndDeferRdf,applyTagsReplaceInFlushAndDeferRdf,applyTagsBatchWithRdf, and the collapsed 4-argapplyTags/applyTagsAdd/applyTagsDeleteoverloads). No references found — no coordination needed.Tests:
Use cases covered
entity/tag/{uuid}/entity/glossaryTerm/{uuid}URI and disappears from the union graph on removal.hasTagtriple (previously orphaned forever).PUT /v1/tags/{id}/assets/{add,remove}andPUT /v1/glossaryTerms/{id}/assets/{add,remove}now syncs RDF (previously no sync at all, most notably on remove).tag/<fqn>synthetic-URI triples appear in the union graph for entity or column tags.Unit tests
RdfUpdater.updateEntity(...)calls wired into already-tested repository methods; per this repo's testing philosophy, the regression protection for RDF-sync behavior belongs in integration tests against the real Fuseki store, not mocks aroundRdfUpdater/RdfRepository.Rdf*Testunit suite (mvn test -pl openmetadata-service -Dtest='Rdf*Test') to confirm no regressions: 0 failures, 0 errors across ~400 tests (RdfPropertyMapperTest,RdfUpdaterTest,RdfUtilsTest,RdfOntologyContractTest, etc.).TaskWorkflowHandlerTest(the one unit test touching a changed file): 11/11 pass.Backend integration tests
openmetadata-integration-tests/.../RdfTagsTierCertificationIT: 8 new tests covering the use cases above. Confirmed RED onmain(special-char-tag round trip crash, bulk add/remove never syncing RDF, column-tag-removal orphan) before this fix, and GREEN after — both runs live against a Docker-backed Fuseki/Postgres/Elasticsearch stack (-DenableRdf=true).TagResourceIT(29 tests),ColumnResourceIT(280 tests, 18 skipped),GlossaryTermResourceIT(59 tests, 1 skipped) in the same run: 0 failures, 0 errors.openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/RdfTagsTierCertificationIT.javaIngestion integration tests
Playwright (UI) tests
Manual testing performed
UI screen recording / screenshots:
Not applicable — no UI changes.
Checklist:
I have read the CONTRIBUTING document.
My PR title is
Fixes <issue-number>: <short explanation>My PR is linked to a GitHub issue via
Fixes #<issue-number>above.I have commented on my code, particularly in hard-to-understand areas.
For JSON Schema changes: I updated the migration scripts or explained why it is not needed. (N/A — no schema changes)
For UI changes: I attached a screen recording and/or screenshots above. (N/A — no UI changes)
I have added tests (unit / integration / Playwright as applicable) and listed them above.
I have added a test that covers the exact scenario we are fixing (
RdfTagsTierCertificationIT, referencing RDF: RdfTagUpdater builds invalid SPARQL for tag FQNs with special characters and writes non-canonical tag triples to the default graph #33474).Opened as draft — final CI run and a maintainer pass over the four commits (tests / core removal / bulk-sync-gap closures / column-orphan fix) still pending before marking ready for review.
🤖 Generated with Claude Code
The PR appears safe to merge; no outstanding correctness, security, or repository-rule issue was identified.
Summary
This PR removes the invalid inline SPARQL tag writer and makes the existing entity-snapshot reconciliation path authoritative for RDF tag state.
TableRepository.addDataModel.Diagram
%%{init: {'theme': 'neutral'}}%% flowchart LR A[Tag or glossary mutation] --> B[Persist tag_usage changes] B --> C[RdfUpdater.updateEntity] C --> D[Capture entity type and ID] D --> E[Post-commit live-write queue] E --> F[Reload authoritative entity state] F --> G[Delete entity-owned RDF snapshot] G --> H[Write canonical entity and column triples] H --> I[Named knowledge graph]Reviews (3) · Last reviewed commit: "Merge branch 'main' into fmcardoso/rdf-r..."