Skip to content

Commit f45bc97

Browse files
ma3uma3u
committed
feat(phase-4d): data plane transfer via DCore with audit logging
- Add GET /fhir/Patient search route to neo4j-proxy (root cause fix) - DCore data plane now successfully proxies to Neo4j Query Proxy - CRO Bayer retrieves 100 FHIR patients via Ed25519 JWT EDR tokens - HDAB BfArM retrieves 2 HealthDCAT-AP datasets via data plane - Add TransferEvent audit logging to all neo4j-proxy endpoints - Create jad/seed-data-transfer.sh automation script - Update Phase 4d progress in planning document Root cause: Data plane baseUrl pointed to /fhir/Patient but only /fhir/Patient/:id/$everything existed. Added base search route. Data flow: Consumer → DCore Data Plane (EDR token) → Neo4j Query Proxy (Cypher) → Neo4j 5-Layer Knowledge Graph → FHIR R4 Bundle / HealthDCAT-AP JSON-LD response
1 parent f5c1bd5 commit f45bc97

3 files changed

Lines changed: 436 additions & 28 deletions

File tree

‎docs/planning-health-dataspace-v2.md‎

Lines changed: 44 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -135,23 +135,23 @@ All three core specifications are now final or near-final:
135135

136136
## Implementation Progress
137137

138-
| Phase | Title | Status | Notes |
139-
| ------ | ------------------------------------------------------ | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
140-
| **1** | Infrastructure Migration (EDC-V + DCore + CFM) | ✅ Complete | 1a–1f all complete; 18 services healthy; 3 tenants + 9 VPAs provisioned; data assets registered; ADR-1–6 accepted |
141-
| **2** | Identity and Trust (DCP v1.0 + Verifiable Credentials) | 🏗️ In progress | 2a complete (DID:web for 3 tenants, Ed25519 keys, all activated — ADR-7); 2b–2c pending |
142-
| **3** | Health Knowledge Graph Layer — Schema & Synthetic Data | ✅ Complete | 5-layer Neo4j schema, EHDS HDAB chain, style sheet |
143-
| **3b** | Real FHIR Data Pipeline (Synthea → Neo4j → OMOP) | ✅ Complete | 167 patients · 5,461 encounters · 2,421 conditions · 37,713 observations · 3,895 drug Rxes · 8,534 procedures |
144-
| **3c** | HealthDCAT-AP Metadata Registration for FHIR Dataset | ✅ Complete | Synthea cohort registered as HealthDCAT-AP catalog entry; 2 distributions + EHDS Art 53 purpose |
145-
| **3d** | README + UI completeness hardening | ✅ Complete | README step order fixed; catalog UI shows datasetType/legalBasis/recordCount |
146-
| **3e** | DSP Marketplace Registration + Compliance Chain | ✅ Complete | Layer 1 DataProduct/Contract/HDABApproval wired to Synthea dataset; compliance UI live dropdowns |
147-
| **3f** | OMOP Research Analytics View | ✅ Complete | Layer 4 cohort dashboard: top conditions/drugs/measurements, gender breakdown, stat cards |
148-
| **3g** | Procedure Pipeline + UI Polish | ✅ Complete | 8,534 Procedure → OMOPProcedureOccurrence; Analytics card on home; 6-stat patient page |
149-
| **3h** | EEHRxF FHIR Profile Alignment | ✅ Complete | EEHRxF category/profile nodes; gap analysis UI; EHDS priority coverage |
150-
| **4** | Dataspace Integration (EDC-V ↔ Neo4j data assets) | 🏗️ In progress | 4a ✅ (assets + policies + contracts); 4b ✅ (3 FINALIZED negotiations + transfer STARTED — ADR-7); 4c ✅ (Federated Catalog: 4 datasets discoverable, HDAB contract FINALIZED); 4d pending |
151-
| **5** | Federated Queries & GraphRAG | 🔲 Not started | Depends on Phase 4 |
152-
| **6a** | Graph Explorer UI (Next.js → Neo4j Bolt) | ✅ Complete | Six views (graph, catalog, compliance, patient, analytics, eehrxf); runs at localhost:3000 |
153-
| **6b** | Full Participant Portal (Aruba + Fraunhofer + Redline) | 🔲 Not started | Depends on Phases 1–4 |
154-
| **7** | TCK DCP & DSP Compliance Verification | 🔲 Not started | Protocol conformance testing; depends on Phases 1–2 |
138+
| Phase | Title | Status | Notes |
139+
| ------ | ------------------------------------------------------ | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
140+
| **1** | Infrastructure Migration (EDC-V + DCore + CFM) | ✅ Complete | 1a–1f all complete; 18 services healthy; 3 tenants + 9 VPAs provisioned; data assets registered; ADR-1–6 accepted |
141+
| **2** | Identity and Trust (DCP v1.0 + Verifiable Credentials) | 🏗️ In progress | 2a complete (DID:web for 3 tenants, Ed25519 keys, all activated — ADR-7); 2b–2c pending |
142+
| **3** | Health Knowledge Graph Layer — Schema & Synthetic Data | ✅ Complete | 5-layer Neo4j schema, EHDS HDAB chain, style sheet |
143+
| **3b** | Real FHIR Data Pipeline (Synthea → Neo4j → OMOP) | ✅ Complete | 167 patients · 5,461 encounters · 2,421 conditions · 37,713 observations · 3,895 drug Rxes · 8,534 procedures |
144+
| **3c** | HealthDCAT-AP Metadata Registration for FHIR Dataset | ✅ Complete | Synthea cohort registered as HealthDCAT-AP catalog entry; 2 distributions + EHDS Art 53 purpose |
145+
| **3d** | README + UI completeness hardening | ✅ Complete | README step order fixed; catalog UI shows datasetType/legalBasis/recordCount |
146+
| **3e** | DSP Marketplace Registration + Compliance Chain | ✅ Complete | Layer 1 DataProduct/Contract/HDABApproval wired to Synthea dataset; compliance UI live dropdowns |
147+
| **3f** | OMOP Research Analytics View | ✅ Complete | Layer 4 cohort dashboard: top conditions/drugs/measurements, gender breakdown, stat cards |
148+
| **3g** | Procedure Pipeline + UI Polish | ✅ Complete | 8,534 Procedure → OMOPProcedureOccurrence; Analytics card on home; 6-stat patient page |
149+
| **3h** | EEHRxF FHIR Profile Alignment | ✅ Complete | EEHRxF category/profile nodes; gap analysis UI; EHDS priority coverage |
150+
| **4** | Dataspace Integration (EDC-V ↔ Neo4j data assets) | 🏗️ In progress | 4a ✅ (assets + policies + contracts); 4b ✅ (3 FINALIZED negotiations + transfer STARTED — ADR-7); 4c ✅ (Federated Catalog: 4 datasets discoverable, HDAB contract FINALIZED); 4d ✅ (Data Plane Transfer: CRO←100 FHIR patients, HDAB←2 HealthDCAT-AP datasets via DCore; audit trail in Neo4j) |
151+
| **5** | Federated Queries & GraphRAG | 🔲 Not started | Depends on Phase 4 |
152+
| **6a** | Graph Explorer UI (Next.js → Neo4j Bolt) | ✅ Complete | Six views (graph, catalog, compliance, patient, analytics, eehrxf); runs at localhost:3000 |
153+
| **6b** | Full Participant Portal (Aruba + Fraunhofer + Redline) | 🔲 Not started | Depends on Phases 1–4 |
154+
| **7** | TCK DCP & DSP Compliance Verification | 🔲 Not started | Protocol conformance testing; depends on Phases 1–2 |
155155

156156
---
157157

@@ -592,18 +592,31 @@ Phase 4 wires the Neo4j health knowledge graph into the live EDC-V data plane, e
592592
> 3. `omop-cohort-statistics` — OMOP CDM Cohort Statistics (Neo4j Layer 4)
593593
> 4. `healthdcatap-catalog` — HealthDCAT-AP Dataset Catalog JSON-LD (Neo4j Layer 2)
594594
595-
#### 4d: Data Plane Transfer via DCore
595+
#### 4d: Data Plane Transfer via DCore ✅
596596

597-
18. Configure DCore Rust data plane for FHIR transfer:
598-
- HTTP push/pull transfer types for FHIR Bundle JSON streaming
599-
- Data Plane Signaling (DPS) for control plane → data plane coordination
600-
- Transfer audit log: `(:TransferProcess)-[:ACCESSED]->(asset)` stored in Neo4j
601-
19. Implement query result proxying:
602-
- CRO sends parameterized Cypher query via DCore HTTP endpoint
603-
- DCore enforces contract-scoped access (only permitted OMOP aggregation queries)
604-
- Query results streamed back through DCore with transfer completion signaling
597+
18. ~~Configure DCore Rust data plane for FHIR transfer:~~
598+
- HTTP pull transfer with Ed25519 JWT bearer tokens (Endpoint Data References)
599+
- Data Plane Signaling (DPS) coordination between control plane and data plane
600+
- Transfer audit log: `(:TransferEvent)-[:ACCESSED]->(HealthDataset)` stored in Neo4j
601+
19. ~~Implement query result proxying:~~
602+
- DCore data plane proxies requests to Neo4j Query Proxy (port 9090)
603+
- Neo4j Query Proxy translates HTTP → Cypher, returns FHIR R4 / HealthDCAT-AP JSON-LD
604+
- Query parameters (`_count`, `gender`, `name`) proxied through data plane
605+
- Added `GET /fhir/Patient` search route to neo4j-proxy for data plane base URL
605606

606-
**Deliverables:** Neo4j assets registered in EDC-V ✅; DSP contract negotiation working end-to-end ✅; Federated Catalog exposing HealthDCAT-AP metadata via DSP ✅; DCore handling FHIR transfers with audit trail.
607+
**Phase 4d Results:**
608+
609+
| Metric | Value |
610+
| ----------------------- | -------------------------------------------------------------- |
611+
| FHIR patients via DCore | 100 (CRO Bayer → data plane → neo4j-proxy → Neo4j) |
612+
| HealthDCAT-AP datasets | 2 (HDAB BfArM → data plane → neo4j-proxy → Neo4j) |
613+
| Auth mechanism | Ed25519 JWT (kid=`dataplane-fhir-private`, alg=`Ed25519`) |
614+
| Data flow | Consumer → DCore Data Plane → Neo4j Query Proxy → Neo4j KG |
615+
| Audit events in Neo4j | TransferEvent nodes with timestamp, endpoint, resultCount |
616+
| Query param proxying | ✅ `_count`, `gender`, `name` forwarded via `proxyQueryParams` |
617+
| Automation script | `jad/seed-data-transfer.sh` |
618+
619+
**Deliverables:** Neo4j assets registered in EDC-V ✅; DSP contract negotiation working end-to-end ✅; Federated Catalog exposing HealthDCAT-AP metadata via DSP ✅; DCore handling FHIR + catalog transfers with audit trail ✅.
607620

608621
> **Root Causes Discovered (Phase 4b):**
609622
>
@@ -612,6 +625,10 @@ Phase 4 wires the Neo4j health knowledge graph into the live EDC-V data plane, e
612625
> 3. Protocol string must be `dataspace-protocol-http:2025-1` (with version suffix), not `dataspace-protocol-http`
613626
> 4. Data plane hostname must match docker-compose service name (`dataplane-fhir`, not `dataplane`)
614627
628+
> **Root Cause Discovered (Phase 4d):**
629+
>
630+
> 5. Neo4j Query Proxy lacked a `GET /fhir/Patient` base route — the data plane's `baseUrl` pointed to `/fhir/Patient` but only `/fhir/Patient/:id/$everything` existed. Added search route returning FHIR R4 searchset Bundle.
631+
615632
### Phase 5: Federated Queries and GraphRAG
616633

617634
15. Deploy two separate Neo4j instances (simulating two HDAB SPEs)

0 commit comments

Comments
 (0)