TECH_DEBT: Add IG validation cost audit tool and cache observability

[c-schuler]

🛠️ Description of Changes

Adds two independent-but-related pieces of validation observability that emerged from team discussion about identifying computationally expensive checks in an IG:

1. validateCode cache hit/miss counters on ValidationCacheService, so we can quantify how much of the terminology cost the existing local cache actually flattens vs. how much still reaches the remote TS.
2. ValidationCostAudit — an internal CLI tool that points at an IG package + sample bundles, drives validation of every resource against every declared profile in isolation, and emits a ranked JSON report of where time is spent and which validation messages fire most often. Gives IG authors an actionable list, not just "profile X is slow."

Together these are "Phase 5a" of the validation-config roadmap discussion - deliberately separate from the SKIP/SUPPRESS categorization work on validation-config, since instrumentation and audit tooling are conceptually distinct from categorization rule migration.

Cache observability (ValidationCacheService)

• Two new LongCounters on ValidationMetrics: link.validation.cache.validate-code.hit, link.validation.cache.validate-code.miss.
• ValidationCacheService.cachedValidateCode(...) bypasses Spring's @Cacheable proxy in favour of an explicit CacheManager.getCache(...).get(key) / put(key, value) path - necessary because the annotation proxy hides the lookup and gives us no way to observe hit vs. miss.
• Behaviour preserved: same Objects.hash(...) key, null results are not cached (matching the previous unless = "#result == null").
• Constructor now takes CacheManager + ValidationMetrics.
• New ValidationCacheServiceTest - three cases: miss populates + increments miss; hit returns cached + no delegate call + increments hit; null result does not poison the cache.

Once this is deployed, hit / (hit + miss) in Grafana gives us a concrete answer to "how much is the cache actually saving us."

Validation cost audit tool (ValidationCostAudit)

Standalone main() class inside the validation module (no new Maven module - deliberate; adding a module was the original plan but the tool reuses production loader patterns cleanly enough that a co-located class is cheaper).

CLI:
mvn -pl validation exec:java
-Dexec.mainClass=com.lantanagroup.link.validation.audit.ValidationCostAudit
-Dexec.args="--ig <path.tgz> --deps

--bundles --iterations 3 --report "

Isolation strategy. For each resource in each bundle, for each profile the resource declares, the tool deep-copies the resource, clears meta.profile, sets it back to the one profile under test, and times a FhirValidator.validateWithResult(...) call. First iteration is warmup and discarded; subsequent iterations are recorded. This attributes cost to a single profile rather than mixing all profiles that would normally fire together.

Terminology chain. In-memory only - DefaultProfileValidationSupport + PrePopulatedValidationSupport (loaded from the primary IG + all deps) + CommonCodeSystemsTerminologyService + InMemoryTerminologyServerValidaingValidationSupport, all wrapped in CachingValidationSupport. Noremote TS - the audit is about slice discrimination and FHIRPath invariants, not network latency, and production's remote-TS caching already flattens the network-cost dimension.

Report shape - JSON, ranked. Key sections:

• byProfile[] - profile URL, sample count, mean/median/p95/max ms, total ms, resource types where the profile fires
• byResourceType[] - same shape rolled up per FHIR resource type
• topMessages[] - validation messages normalized (quoted strings, UUIDs, array indexes, bare numbers collapsed to *), deduped, ranked by count, each with three example locations and three example resources
• bundles[].samples[] - full per-sample records for drill-down, including per-sample message list on the first recorded iteration

Executive summary printed to stdout so runs give instant feedback without opening the JSON.

Log noise. Silences HAPI's INFO-level chatter (Fetching CodeSystem for..., Loading structure definitions from...) by default; --verbose / -v restores it. WARN/ERROR always get through.

Sample output

Ran against gov.cdc.nhsn.measures.r4@2.0.0-cibuild with 12 dependency IGs and both NHSN acute-care bundles (166 samples, ~4 s wall time):

Top profiles by total time:

• 1005.7 ms (n=64, mean=15.7, p95=27.4) hl7.fhir/StructureDefinition/shareablevalueset
• 984.7 ms (n=64, mean=15.4, p95=24.8) hl7.fhir/uv/crmi/StructureDefinition/crmi-publishablevalueset - 766.0 ms (n=4, mean=191.5, p95=266.5) hl7.fhir/us/cqfmeasures/StructureDefinition/cohort-measure-cqfm
• 673.3 ms (n=4, mean=168.3, p95=198.1) hl7.fhir/us/cqfmeasures/StructureDefinition/computable-measure-cqfm

Top validation messages (normalized, deduped):

• x90 ERROR Unknown code 'http://hl7.org/fhir/expression-language#text/cql-identifier'
• x78 WARNING Constraint failed: dom-6: 'A resource should have narrative for robust management' - x66 ERROR Extension 'valueset-effectiveDate' allows for types [dateTime] but found type ...
• x32 ERROR ValueSet.description: minimum required = 1, but only found 0 (shareablevalueset) - x32 ERROR ValueSet.experimental: minimum required = 1, but only found 0 (shareablevalueset)

Reading the numbers: Measure resources are the per-call hotspot (~180 by volume (146 samples). Much of the shareablevalueset cost is HAPIreporting the same missing-required-field errors — fix those in the IG and per-ValueSet mean should drop. The cohort-measure-cqfm per-call cost (192 ms) doesn't obviously trace to the top messages, which points at slice discriminas the next investigation.

Documentation

Java/validation/COST-AUDIT.md (228 lines) - invocation, CLI flags, iny, terminology chain rationale, JSON report shape, interpretationguide, explicit caveats (no per-invariant timing, warm-cache only, per-entry vs. bundle-level), NHSN case study with real numbers. README.md gets a one-line "Utilities" pointer.

What is NOT in this PR

• Per-invariant / per-constraint timing. HAPI doesn't expose this natively. If the audit points at a profile and the top messages don't obviously explain the cost, follow-up work is instrumenting the FHIRPath engine.
• Per-profile histograms in production ValidationMetrics. Originally scoped as a separate "piece 2" step, but production traffic is almost entirely Bundle at the top level, so a per-profile production histogram would emit one bucket and give the same signal as link.validation.validate.duration. The interesting per-profile split only exists when you iterate resources, which is exactly what the audit driver does. Rolling the histograms into the driver kept the diff honest.
• CI job that runs the audit tool. It's intentionally an on-demand utility - no scheduled job, no artifact published from it.
• Unit tests for ValidationCostAudit. The tool is internal and single end-to-end against the NHSN IG. If reviewers want a smoke test that runs against a fixture bundle in CI, happy to add one.

Test plan

• CI Java unit tests pass (adds ValidationCacheServiceTest, existing tests still green)
• Audit tool ran end-to-end against gov.cdc.nhsn.measures.r4@2.0.0-cibuild and produced coherent output
• After deploy, confirm Grafana shows link.validation.cache.validate-code.hit and .miss streams within minutes of traffic
• Sanity check that hit rate matches the caching claim we grounded this in - expect high hit rate (>90%) on steady-state traffic

🧪 Testing Performed

Audit tool ran end-to-end against gov.cdc.nhsn.measures.r4@2.0.nt output

🧑‍🔬 Unit Testing

• I have written or updated unit tests to cover my changes
• Coverage: 5.2%

📓 Documentation Updated

Java/validation/COST-AUDIT.md

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 3007d5c2-f86f-4f7a-8d43-da46c63e074d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch validation-cost-audit

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}