|
| 1 | +--- |
| 2 | +doc_type: spec |
| 3 | +status: accepted |
| 4 | +date: 2026-06-21 |
| 5 | +owner: acumenus |
| 6 | +module: ingestion |
| 7 | +lineage_anchor: false |
| 8 | +supersedes: [] |
| 9 | +superseded_by: null |
| 10 | +related_prs: [] |
| 11 | +related_code: |
| 12 | + - backend/app/Services/Fhir/FhirBulkMapper.php |
| 13 | + - backend/app/Services/Fhir/FhirNdjsonProcessorService.php |
| 14 | + - backend/app/Services/Fhir/CrosswalkService.php |
| 15 | + - backend/app/Jobs/Fhir/RunFhirSyncJob.php |
| 16 | + - backend/app/Models/App/FhirConnection.php |
| 17 | +--- |
| 18 | + |
| 19 | +# FHIR Ingestion — Medgnosis Parity Port → OMOP CDM (Design Spec) |
| 20 | + |
| 21 | +## Goal |
| 22 | + |
| 23 | +Bring Parthenon's **inbound FHIR ingestion** to parity with the Medgnosis |
| 24 | +`feature/fhir-edw-ingestion-expansion` work — adapted from Medgnosis's `phm_edw` |
| 25 | +target to Parthenon's **OMOP CDM v5.4** target. Specifically: add 6 clinical |
| 26 | +resource types + reference dimensions, add soft-delete / `entered-in-error` / |
| 27 | +Bulk-`deleted`-manifest handling, and validate end-to-end against a live EHR |
| 28 | +sandbox. |
| 29 | + |
| 30 | +## Context — what already exists in Parthenon |
| 31 | + |
| 32 | +Parthenon's inbound pipeline already mirrors Medgnosis's architecture: |
| 33 | + |
| 34 | +- `RunFhirSyncJob` — SMART Backend Services (JWT) auth + Bulk `$export` pull from |
| 35 | + an external EHR defined by a `FhirConnection` (`fhir_base_url`, `token_endpoint`, |
| 36 | + `client_id`, `export_resource_types`). |
| 37 | +- `FhirNdjsonProcessorService` — two-pass, crosswalk-keyed hydration: Pass 1 |
| 38 | + Patient + Encounter → `person` / `visit_occurrence` + crosswalks; Pass 2 clinical |
| 39 | + resources resolved against the crosswalks. |
| 40 | +- `FhirBulkMapper` (855 lines) — `mapResource()` dispatches a `match` on |
| 41 | + `resourceType` to inline `map*` methods, each returning |
| 42 | + `list<{cdm_table, data, fhir_resource_type, fhir_resource_id}>`. Currently covers |
| 43 | + ~10 types (Patient, Encounter, Condition, Observation, Procedure, Medication*, |
| 44 | + Immunization, AllergyIntolerance, DiagnosticReport). |
| 45 | +- Crosswalk tables already present: `fhir_patient_crosswalk`, |
| 46 | + `fhir_encounter_crosswalk`, `fhir_provider_crosswalk`, `fhir_caresite_crosswalk`, |
| 47 | + `fhir_location_crosswalk` (so reference-dimension plumbing partly exists; |
| 48 | + `mapEncounter` already emits provider/location/care_site rows). |
| 49 | + |
| 50 | +**Not present today:** the 6 expansion resource types, soft-delete / |
| 51 | +`entered-in-error`, and the Bulk `deleted`-manifest path. |
| 52 | + |
| 53 | +## Architecture |
| 54 | + |
| 55 | +Introduce a small, focused `ResourceMapper` seam rather than growing the |
| 56 | +already-oversized `FhirBulkMapper`: |
| 57 | + |
| 58 | +- `interface ResourceMapper { public function resourceType(): string; public function map(array $resource, string $siteKey): array; }` |
| 59 | + returning the same `list<{cdm_table, data, fhir_resource_type, fhir_resource_id}>` |
| 60 | + contract. |
| 61 | +- One class per new resource under `backend/app/Services/Fhir/Mappers/`: |
| 62 | + `DocumentReferenceMapper`, `CoverageMapper`, `ServiceRequestMapper`, |
| 63 | + `CarePlanMapper`, `GoalMapper`, `CareTeamMapper`. Reference-dimension |
| 64 | + resolution (Practitioner/Organization/Location) goes through a shared |
| 65 | + `ReferenceDimensionResolver` that wraps the existing provider/care_site/location |
| 66 | + crosswalks. |
| 67 | +- `FhirBulkMapper::mapResource()` falls through its existing `match` to a |
| 68 | + registry lookup for the new types — **existing inline mappers are unchanged** |
| 69 | + (no big-bang refactor of the 855-line file; the registry is the extension point |
| 70 | + for all future resources). |
| 71 | + |
| 72 | +This keeps each new unit independently testable with one clear purpose. |
| 73 | + |
| 74 | +## Resource → OMOP mapping (hybrid: standard tables + 3 extension tables) |
| 75 | + |
| 76 | +| FHIR | OMOP target | Key fields | |
| 77 | +|---|---|---| |
| 78 | +| DocumentReference | `note` | `note_text` from inline `content.attachment.data`/`url` stub; `note_type_concept_id`/`note_class_concept_id` from `type`; `note_date` from `date`; `person_id` via patient crosswalk; `visit_occurrence_id` via `context.encounter`; `provider_id` via author. | |
| 79 | +| Coverage | `payer_plan_period` | `payer_plan_period_start_date`/`end_date` from `period`; `payer_source_value` from `payor`; `plan_source_value` from `class[type=plan]`; person via subscriber/beneficiary crosswalk. | |
| 80 | +| ServiceRequest | `procedure_occurrence` (status-gated) | only `status ∈ {active, completed}` and `intent ∈ {order, original-order}`; `procedure_concept_id` from `code`; `procedure_date` from `authoredOn`/`occurrence`; `procedure_type_concept_id` = an "EHR order list entry" concept; requester→`provider_id`; `encounter`→`visit_occurrence_id`. Draft/revoked/cancelled are skipped (not errors). | |
| 81 | +| Practitioner | `provider` (existing crosswalk) | reference dim; get-or-create, backfill `provider_id` FKs. | |
| 82 | +| Organization | `care_site` (existing crosswalk) | reference dim. | |
| 83 | +| Location | `location` / `care_site` (existing crosswalk) | reference dim. | |
| 84 | +| CarePlan | **extension `care_plan`** | id, person, period, status, intent, category, encounter, addresses (conditions). | |
| 85 | +| Goal | **extension `care_goal`** | id, person, lifecycle status, achievement, description, target, `care_plan_id` link (synthetic plan if standalone). | |
| 86 | +| CareTeam | **extension `care_team` + `care_team_member`** | id, person, period, status, participants (role + provider/org via crosswalk). | |
| 87 | + |
| 88 | +### Extension tables |
| 89 | + |
| 90 | +Three new tables follow Parthenon's existing OMOP-extension-bridge convention |
| 91 | +(per-CDM-schema, FK to `person`/`visit_occurrence`, OMOP-style `*_concept_id` + |
| 92 | +`*_source_value` columns), created by Laravel migrations: |
| 93 | + |
| 94 | +- `care_plan` (PK `care_plan_id`, `person_id`, `care_plan_start_date`, |
| 95 | + `care_plan_end_date`, `status_concept_id`, `intent_concept_id`, |
| 96 | + `category_concept_id`, `visit_occurrence_id`, `care_plan_source_value`, |
| 97 | + `care_plan_source_concept_id`). |
| 98 | +- `care_goal` (PK `care_goal_id`, `person_id`, `care_plan_id` (nullable FK), |
| 99 | + `lifecycle_status`, `achievement_status_concept_id`, `goal_start_date`, |
| 100 | + `goal_source_value`, `goal_source_concept_id`). |
| 101 | +- `care_team` (PK `care_team_id`, `person_id`, `care_team_start_date`, |
| 102 | + `care_team_end_date`, `status`, `care_team_source_value`) + |
| 103 | + `care_team_member` (PK, `care_team_id` FK, `provider_id` (nullable), |
| 104 | + `care_site_id` (nullable), `role_concept_id`, `role_source_value`). |
| 105 | + |
| 106 | +The exact NOT-NULL/concept columns are finalized against the live OMOP schema in |
| 107 | +the implementation plan; the migration adds them to each writable CDM schema the |
| 108 | +ingestion targets. |
| 109 | + |
| 110 | +## Soft-delete / entered-in-error / Bulk `deleted` manifest |
| 111 | + |
| 112 | +OMOP CDM is append-only and has no soft-delete column, so the OMOP-correct |
| 113 | +behavior is to **remove the erroneous row(s)** while preserving an audit trail in |
| 114 | +the crosswalk + sync-run metadata (vs Medgnosis's `active_ind='N'`): |
| 115 | + |
| 116 | +- **During hydration:** a resource arriving with `status = entered-in-error` (or |
| 117 | + `Observation.status`/`Condition.verificationStatus = entered-in-error`) → |
| 118 | + `CrosswalkService::deleteByResource(siteKey, resourceType, resourceId)` resolves |
| 119 | + the crosswalk row → deletes the mapped CDM row by `(cdm_table, cdm_id)` → |
| 120 | + stamps the crosswalk (`deleted_at`, `deleted_reason`). The resource is **not** |
| 121 | + also hydrated. |
| 122 | +- **Bulk `deleted` output:** `$export` emits a `deleted` output set (NDJSON of |
| 123 | + FHIR Bundles whose entries carry `request.method = DELETE` / |
| 124 | + `request.url = ResourceType/id`). A new `processBulkDeletions()` step in the sync |
| 125 | + job downloads each `deleted` file, extracts the deleted references, and runs the |
| 126 | + same crosswalk-resolve-and-delete path. Per-file fetch errors are counted, not |
| 127 | + fatal; results recorded in the `FhirSyncRun` metadata. |
| 128 | +- Crosswalk migration adds `deleted_at` / `deleted_reason` columns where absent. |
| 129 | + |
| 130 | +## Live-EHR end-to-end validation |
| 131 | + |
| 132 | +- A `FhirConnection` is configured for the EHR sandbox using the provided |
| 133 | + **non-prod `client_id` `66b2fa2f-52c2-4a1e-90a8-3142a4913a79`**, the sandbox |
| 134 | + `fhir_base_url` + `token_endpoint`, and a key pair whose public key is registered |
| 135 | + to that client (SMART Backend Services `private_key_jwt`). The client_id and key |
| 136 | + live in the connection/secret config — **never hardcoded**. |
| 137 | +- `export_resource_types` is extended to the full 15-type set so the kickoff |
| 138 | + requests the new resources (`$export?_type=…`). |
| 139 | +- Validation: run a real Bulk `$export`, hydrate, and verify the new OMOP rows |
| 140 | + (counts + spot-checks per resource) and a `deleted`-manifest soft-delete. |
| 141 | + |
| 142 | +## Test methodology (3 levels, per resource — Medgnosis's "double-check everything") |
| 143 | + |
| 144 | +1. **Pest mock tests** — assert the emitted CDM rows/columns for each mapper |
| 145 | + (insert + update + skip paths), mocking the DB. |
| 146 | +2. **Real-schema rollback** — execute each new INSERT/UPDATE inside |
| 147 | + `BEGIN…ROLLBACK` against the live OMOP **test** schema (zero data change) to |
| 148 | + catch column / NOT-NULL / cast errors mocks cannot. |
| 149 | +3. **Live `$export` end-to-end** — the sandbox run above. |
| 150 | + |
| 151 | +Every code phase runs the pre-commit gate (Pint, PHPStan L8, tsc/ESLint where |
| 152 | +touched) and the bounded Pest lanes before it lands. |
| 153 | + |
| 154 | +## Phasing |
| 155 | + |
| 156 | +1. **Foundations** — extension migrations (`care_plan`/`care_goal`/`care_team`/ |
| 157 | + `care_team_member`) + crosswalk `deleted_at`/`deleted_reason`; the |
| 158 | + `ResourceMapper` interface + registry wired into `FhirBulkMapper`; the |
| 159 | + `ReferenceDimensionResolver`. |
| 160 | +2. **Reference dimensions** — Practitioner→`provider`, Organization→`care_site`, |
| 161 | + Location→`location`/`care_site` get-or-create + FK backfill (extending the |
| 162 | + existing crosswalk usage). |
| 163 | +3. **Clean-home mappers** — DocumentReference→`note`, Coverage→`payer_plan_period`, |
| 164 | + ServiceRequest→`procedure_occurrence`. |
| 165 | +4. **Extension mappers** — CarePlan, Goal, CareTeam (+ members). |
| 166 | +5. **Soft-delete** — `entered-in-error` hydration path + Bulk `deleted` manifest. |
| 167 | +6. **Wire + validate** — extend `$export _type` + connection defaults; live-EHR |
| 168 | + end-to-end with the sandbox client; closeout devlog. |
| 169 | + |
| 170 | +## Acceptance |
| 171 | + |
| 172 | +- Inbound clinical resource coverage **+6** types (DocumentReference, Coverage, |
| 173 | + ServiceRequest, CarePlan, Goal, CareTeam) into OMOP CDM, **plus** |
| 174 | + Practitioner/Organization/Location reference dimensions — each with insert/update |
| 175 | + + entered-in-error delete semantics, verified by Pest mocks + real-schema rollback. |
| 176 | +- Bulk `deleted` manifest processed; soft-deletes audited in crosswalk + run |
| 177 | + metadata. |
| 178 | +- A live sandbox `$export` (via the provided client) ingests the new resource |
| 179 | + types and the new OMOP rows are verified. |
| 180 | +- The "FHIR Bulk Data reader" / inbound-ingestion completeness item in the |
| 181 | + ingestion-templates Phase-4 + completion plans is closed with this evidence. |
| 182 | + |
| 183 | +## Risks / open items |
| 184 | + |
| 185 | +- **ServiceRequest→`procedure_occurrence`** is a known OHDSI compromise (an order |
| 186 | + is not a performed procedure); gated to `active/completed` + an "order" type |
| 187 | + concept, and revisited if it pollutes incidence analytics. |
| 188 | +- **Extension table concept_ids** require vocabulary lookups (status/intent/role); |
| 189 | + where no standard concept exists, store `*_source_value` + concept_id 0. |
| 190 | +- **Sandbox key registration** — the provided client_id needs its public key |
| 191 | + registered with the sandbox (Workstream prerequisite for Phase 6). |
| 192 | +- Exact extension-table column nullability is finalized against the live OMOP |
| 193 | + schema during Phase 1. |
0 commit comments