Skip to content

Commit 66fbf9d

Browse files
committed
feat(phase4b): contract negotiation + data transfer via DSP 2025-1
Phase 4b complete - full Dataspace Protocol flow working end-to-end: CRO to Clinic catalog discovery, contract negotiation (FINALIZED), HttpData-PULL transfer (STARTED) New: jad/seed-contract-negotiation.sh - v5alpha management API for negotiation + transfer - v1alpha virtual API for catalog discovery - Data plane registration with correct Docker hostnames - Polling loops for negotiation FINALIZED + transfer STARTED states - HDAB operator catalog oversight verification Updated: jad/seed-data-assets.sh - Added Step 5: participant context activation (CREATED to ACTIVATED) - Added Step 6: data plane registration (dataplane-fhir:8083) Updated: docs/planning-health-dataspace-v2.md - Phase 4b marked complete with root causes documented - Progress table updated (4a + 4b complete, 4c-4d pending) Root causes discovered and fixed: 1. operandLeft must use full URI (https://w3id.org/edc/v0.0.1/ns/id) 2. Participant contexts need ACTIVATED state (300) for DID serving 3. Protocol string requires version suffix (dataspace-protocol-http:2025-1) 4. Data plane URL must use docker-compose service name (dataplane-fhir)
1 parent 5b427da commit 66fbf9d

4 files changed

Lines changed: 523 additions & 38 deletions

File tree

.gitignore

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -44,8 +44,7 @@ docker-compose.override.yml
4444
vendor/
4545

4646
# Connector — custom EDC connector build artifacts
47-
connector/.gradle/
48-
connector/*/build/
47+
connector/
4948

5049
# Synthea — generated patient data (never commit)
5150
neo4j/import/fhir/*.json

docs/planning-health-dataspace-v2.md

Lines changed: 40 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -127,29 +127,29 @@ All three core specifications are now final or near-final:
127127

128128
## Implementation Progress
129129

130-
| Phase | Title | Status | Notes |
131-
| ------ | ------------------------------------------------------ | -------------- | ------------------------------------------------------------------------------------------------------------------------ |
132-
| **1** | Infrastructure Migration (EDC-V + DCore + CFM) | 🏗️ In progress | 1a deployed via Docker Compose (18 services healthy); 1c+1d+1e+1f complete; ADR-1–6 accepted; 1b (tenant config) pending |
133-
| **2** | Identity and Trust (DCP v1.0 + Verifiable Credentials) | 🏗️ In progress | IssuerService + IdentityHub running; seed tenant + attestation/credential defs created; DID:web pending |
134-
| **3** | Health Knowledge Graph Layer — Schema & Synthetic Data | ✅ Complete | 5-layer Neo4j schema, EHDS HDAB chain, style sheet |
135-
| **3b** | Real FHIR Data Pipeline (Synthea → Neo4j → OMOP) | ✅ Complete | 127 patients · 3,031 encounters · 1,045 conditions · 19,195 observations · 2,232 drug Rxes |
136-
| **3c** | HealthDCAT-AP Metadata Registration for FHIR Dataset | ✅ Complete | Synthea cohort registered as HealthDCAT-AP catalog entry; 2 distributions + EHDS Art 53 purpose |
137-
| **3d** | README + UI completeness hardening | ✅ Complete | README step order fixed; catalog UI shows datasetType/legalBasis/recordCount |
138-
| **3e** | DSP Marketplace Registration + Compliance Chain | ✅ Complete | Layer 1 DataProduct/Contract/HDABApproval wired to Synthea dataset; compliance UI live dropdowns |
139-
| **3f** | OMOP Research Analytics View | ✅ Complete | Layer 4 cohort dashboard: top conditions/drugs/measurements, gender breakdown, stat cards |
140-
| **3g** | Procedure Pipeline + UI Polish | ✅ Complete | 8,534 Procedure → OMOPProcedureOccurrence; Analytics card on home; 6-stat patient page |
141-
| **3h** | EEHRxF FHIR Profile Alignment | ✅ Complete | EEHRxF category/profile nodes; gap analysis UI; EHDS priority coverage |
142-
| **4** | Dataspace Integration (EDC-V ↔ Neo4j data assets) | 🔲 Not started | Depends on Phases 1, 2, 3c; 4a asset payloads prepared |
143-
| **5** | Federated Queries & GraphRAG | 🔲 Not started | Depends on Phase 4 |
144-
| **6a** | Graph Explorer UI (Next.js → Neo4j Bolt) | ✅ Complete | Four views; runs at localhost:3000 |
145-
| **6b** | Full Participant Portal (Aruba + Fraunhofer + Redline) | 🔲 Not started | Depends on Phases 1–4 |
146-
| **7** | TCK DCP & DSP Compliance Verification | 🔲 Not started | Protocol conformance testing; depends on Phases 1–2 |
130+
| Phase | Title | Status | Notes |
131+
| ------ | ------------------------------------------------------ | -------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
132+
| **1** | Infrastructure Migration (EDC-V + DCore + CFM) | ✅ Complete | 1a–1f all complete; 18 services healthy; 3 tenants + 9 VPAs provisioned; data assets registered; ADR-1–6 accepted |
133+
| **2** | Identity and Trust (DCP v1.0 + Verifiable Credentials) | 🏗️ In progress | IssuerService + IdentityHub running; seed tenant + attestation/credential defs created; DID:web pending |
134+
| **3** | Health Knowledge Graph Layer — Schema & Synthetic Data | ✅ Complete | 5-layer Neo4j schema, EHDS HDAB chain, style sheet |
135+
| **3b** | Real FHIR Data Pipeline (Synthea → Neo4j → OMOP) | ✅ Complete | 167 patients · 5,461 encounters · 2,421 conditions · 37,713 observations · 3,895 drug Rxes · 8,534 procedures |
136+
| **3c** | HealthDCAT-AP Metadata Registration for FHIR Dataset | ✅ Complete | Synthea cohort registered as HealthDCAT-AP catalog entry; 2 distributions + EHDS Art 53 purpose |
137+
| **3d** | README + UI completeness hardening | ✅ Complete | README step order fixed; catalog UI shows datasetType/legalBasis/recordCount |
138+
| **3e** | DSP Marketplace Registration + Compliance Chain | ✅ Complete | Layer 1 DataProduct/Contract/HDABApproval wired to Synthea dataset; compliance UI live dropdowns |
139+
| **3f** | OMOP Research Analytics View | ✅ Complete | Layer 4 cohort dashboard: top conditions/drugs/measurements, gender breakdown, stat cards |
140+
| **3g** | Procedure Pipeline + UI Polish | ✅ Complete | 8,534 Procedure → OMOPProcedureOccurrence; Analytics card on home; 6-stat patient page |
141+
| **3h** | EEHRxF FHIR Profile Alignment | ✅ Complete | EEHRxF category/profile nodes; gap analysis UI; EHDS priority coverage |
142+
| **4** | Dataspace Integration (EDC-V ↔ Neo4j data assets) | 🏗️ In progress | 4a complete (assets + policies + contracts + data planes); 4b complete (DSP negotiation FINALIZED + transfer STARTED); 4c–4d pending |
143+
| **5** | Federated Queries & GraphRAG | 🔲 Not started | Depends on Phase 4 |
144+
| **6a** | Graph Explorer UI (Next.js → Neo4j Bolt) | ✅ Complete | Six views (graph, catalog, compliance, patient, analytics, eehrxf); runs at localhost:3000 |
145+
| **6b** | Full Participant Portal (Aruba + Fraunhofer + Redline) | 🔲 Not started | Depends on Phases 1–4 |
146+
| **7** | TCK DCP & DSP Compliance Verification | 🔲 Not started | Protocol conformance testing; depends on Phases 1–2 |
147147

148148
---
149149

150150
## Implementation Roadmap
151151

152-
### Phase 1: Infrastructure Migration (JAD-Based)
152+
### Phase 1: Infrastructure Migration (JAD-Based)
153153

154154
Phase 1 bootstraps the full EDC-V + DCore + CFM stack using the [JAD (Joint Architecture Demo)](https://github.com/Metaform/jad) as the reference deployment. JAD provides pre-built container images, Kubernetes manifests, and automated end-to-end tests — we adapt its infrastructure to serve the health dataspace domain.
155155

@@ -329,17 +329,18 @@ Scripts in `scripts/` automate the full pipeline:
329329
- Creates: `OMOPPerson`, `OMOPVisitOccurrence`, `OMOPConditionOccurrence`, `OMOPMeasurement`, `OMOPDrugExposure`
330330
- Adds `MAPPED_TO` (FHIR → OMOP) and `CODED_BY` (OMOP → SNOMED/LOINC/RxNorm) relationships
331331

332-
**Current graph state (50-patient Synthea cohort, Massachusetts):**
332+
**Current graph state (100-patient Synthea cohort, Massachusetts):**
333333

334-
| Layer 3 FHIR | Count | Layer 4 OMOP | Count |
335-
| ----------------- | ------ | ----------------------- | ----- |
336-
| Patient | 127\* | OMOPPerson | 127 |
337-
| Encounter | 3,031 | OMOPVisitOccurrence | 3,031 |
338-
| Condition | 1,045 | OMOPConditionOccurrence | 1,045 |
339-
| Observation | 19,195 | OMOPMeasurement | 737 |
340-
| MedicationRequest | 2,232 | OMOPDrugExposure | 2,232 |
334+
| Layer 3 FHIR | Count | Layer 4 OMOP | Count |
335+
| ----------------- | ------ | ----------------------- | ------ |
336+
| Patient | 167\* | OMOPPerson | 167 |
337+
| Encounter | 5,461 | OMOPVisitOccurrence | 5,461 |
338+
| Condition | 2,421 | OMOPConditionOccurrence | 2,421 |
339+
| Observation | 37,713 | OMOPMeasurement | 34,203 |
340+
| MedicationRequest | 3,895 | OMOPDrugExposure | 3,895 |
341+
| Procedure | 8,534 | OMOPProcedureOccurrence | 8,534 |
341342

342-
_\* 127 includes deceased patients generated by Synthea alongside the 50 living target patients._
343+
_\* 167 includes deceased patients generated by Synthea alongside the 100 living target patients._
343344

344345
The Graph Explorer UI (`/graph` and `/patient`) immediately reflects the real patient data.
345346

@@ -348,7 +349,7 @@ The Graph Explorer UI (`/graph` and `/patient`) immediately reflects the real pa
348349
The Synthea cohort loaded in Phase 3b needs a corresponding **Layer 2** catalog entry so EDC-V can expose it as a discoverable data asset. This is implemented as an idempotent Cypher script:
349350

350351
- `neo4j/register-fhir-dataset-hdcatap.cypher` — creates/updates the `HealthDataset` node with full HealthDCAT-AP properties (title, description, publisher, temporal coverage, spatial coverage, themes, access conditions)
351-
- Links the dataset to all 127 `Patient` nodes via `FROM_DATASET`
352+
- Links the dataset to all 167 `Patient` nodes via `FROM_DATASET`
352353
- Registers `Distribution` nodes (Bolt + REST + DCore HTTP endpoints) so EDC-V can reference the access URL
353354
- Adds EHDS purpose restriction annotation (Article 53 permitted purposes)
354355

@@ -513,11 +514,11 @@ The **European Electronic Health Record Exchange Format (EEHRxF)** was establish
513514
- EHDS implementation timeline visualization (2025 → 2031)
514515
- Gap analysis highlighting missing resources (e.g., DiagnosticReport, ImagingStudy)
515516

516-
### Phase 4: Dataspace Integration (EDC-V ↔ Neo4j)
517+
### Phase 4: Dataspace Integration (EDC-V ↔ Neo4j) 🏗️
517518

518519
Phase 4 wires the Neo4j health knowledge graph into the live EDC-V data plane, enabling full DSP contract negotiation and credentialed data access.
519520

520-
#### 4a: Data Asset Registration
521+
#### 4a: Data Asset Registration
521522

522523
12. Register Neo4j data assets on the Clinic's EDC-V instance via the generated TypeScript Admin API client:
523524
- **FHIR Cohort Asset** — Cypher query endpoint returning FHIR R4 patient bundles
@@ -529,7 +530,7 @@ Phase 4 wires the Neo4j health knowledge graph into the live EDC-V data plane, e
529530
- Anonymization requirements (k-anonymity threshold for cohort queries)
530531
- Re-identification prohibition (as currently modeled in `OdrlPolicy` graph nodes)
531532

532-
#### 4b: Contract Negotiation Flow
533+
#### 4b: Contract Negotiation Flow
533534

534535
14. Implement end-to-end DSP contract negotiation:
535536
- CRO discovers FHIR Cohort Asset via HDAB Federated Catalog
@@ -561,7 +562,14 @@ Phase 4 wires the Neo4j health knowledge graph into the live EDC-V data plane, e
561562
- DCore enforces contract-scoped access (only permitted OMOP aggregation queries)
562563
- Query results streamed back through DCore with transfer completion signaling
563564

564-
**Deliverables:** Neo4j assets registered in EDC-V; DSP contract negotiation working end-to-end; Federated Catalog exposing HealthDCAT-AP metadata; DCore handling FHIR transfers with audit trail.
565+
**Deliverables:** Neo4j assets registered in EDC-V ✅; DSP contract negotiation working end-to-end ✅; Federated Catalog exposing HealthDCAT-AP metadata; DCore handling FHIR transfers with audit trail.
566+
567+
> **Root Causes Discovered (Phase 4b):**
568+
>
569+
> 1. Contract definition `operandLeft` must use full URI `https://w3id.org/edc/v0.0.1/ns/id`, not `@id`
570+
> 2. Participant contexts must be ACTIVATED (state=300) for DID document serving; CFM creates them as CREATED (200)
571+
> 3. Protocol string must be `dataspace-protocol-http:2025-1` (with version suffix), not `dataspace-protocol-http`
572+
> 4. Data plane hostname must match docker-compose service name (`dataplane-fhir`, not `dataplane`)
565573
566574
### Phase 5: Federated Queries and GraphRAG
567575

0 commit comments

Comments
 (0)