|
| 1 | +# Warehouse ↔ DataFoundry integration contract |
| 2 | + |
| 3 | +> This is the producer-side contract for **DataFoundry** (the Databricks lakehouse project that |
| 4 | +> consumes this data) — what actually lands in S3 today, in what shape, at what path, with what |
| 5 | +> guarantees. For the *why* behind each design choice (cloud-agnostic CDC, Auto Loader vs. |
| 6 | +> alternatives, Silver/Gold design, the metrics catalog) see [`BI_Plan.md`](BI_Plan.md) — this file |
| 7 | +> only states what DataFoundry can rely on right now. |
| 8 | +> |
| 9 | +> Nothing below has been deployed to a live AWS environment yet (`BI_LAKE_BUCKET` has never been set |
| 10 | +> on a real `terraform apply`) — every guarantee here has been verified locally against real |
| 11 | +> Postgres/Debezium/RabbitMQ/S3-compatible (LocalStack) containers, not the real thing. Status per |
| 12 | +> row is called out explicitly; nothing is glossed over as "done" that hasn't actually run. |
| 13 | +
|
| 14 | +## 1. The landing zone |
| 15 | + |
| 16 | +One S3 bucket, on/off with a single Terraform variable: |
| 17 | + |
| 18 | +- **Bucket**: `warehouse-bi-lake-<account-id>` (override via `bi_lake_bucket_name`, |
| 19 | + `infra/terraform/bootstrap/variables.tf`) |
| 20 | +- **Switch**: setting the `BI_LAKE_BUCKET` repo variable turns on *both* producer paths below at |
| 21 | + once (`docs/deploy.md`) — there's no way to enable one without the other today. |
| 22 | +- **IAM**: a single shared ECS task role can `s3:PutObject` under this bucket's root; nothing else |
| 23 | + writes to it. |
| 24 | + |
| 25 | +Three independent writers land into the same bucket, under three separate prefixes. DataFoundry's |
| 26 | +Auto Loader jobs should point at prefixes, not the bucket root. |
| 27 | + |
| 28 | +| Prefix | Writer | Format | Trigger | |
| 29 | +|---|---|---|---| |
| 30 | +| `bronze/<source_system>/<event_type>/date=YYYY-MM-DD/<event_id>.parquet` | `S3ParquetBronzeExportWriter` (`Warehouse.ServiceDefaults.BiExport`) — one instance per backend service | Parquet, one row per file | Each service's own published integration events, via a dedicated `<service>.bi-export` RabbitMQ queue | |
| 31 | +| `bronze-cdc/<schema>/<table>/date=YYYY-MM-DD/<lsn>.json` | `Warehouse.CdcBridge` (one shared worker) | Raw Debezium change-event JSON (`before`/`after`/`op`/`source`), byte-for-byte, schemas stripped | Postgres logical replication → Debezium Server → RabbitMQ → CdcBridge, for internal state that never becomes an integration event | |
| 32 | +| `bronze/identity-service/UserSnapshot/date=YYYY-MM-DD/<event_id>.parquet` | `Warehouse.IdentityBridge` (reuses the same Parquet writer) | Parquet, one row per user per poll | Periodic poll of Keycloak's Admin REST API (no CDC/events possible — Keycloak dev-mode has no persistent DB) | |
| 33 | + |
| 34 | +**Idempotency**: every key is deterministic — the event-driven path keys on the Wolverine outbox |
| 35 | +message id, CDC keys on the Postgres LSN (the WAL position of the change). A redelivery from the |
| 36 | +at-least-once outbox, or from Debezium, overwrites the same S3 object rather than producing a |
| 37 | +duplicate row. DataFoundry's ingestion does **not** need its own dedup step for either path. |
| 38 | + |
| 39 | +**Schema-on-read**: every Parquet column (`bronze/...`) is written as a string, including numbers and |
| 40 | +timestamps (ISO-8601, `"O"` format) — typing happens in Silver, not Bronze, by design. The CDC path |
| 41 | +is native JSON with no imposed schema at all; Debezium's own envelope carries the typed `before`/ |
| 42 | +`after` row images. |
| 43 | + |
| 44 | +## 2. Common envelope (event-driven path only) |
| 45 | + |
| 46 | +Every `bronze/...` Parquet file carries these columns before the event's own fields: |
| 47 | + |
| 48 | +``` |
| 49 | +_source_system : string // "warehouse-service" | "logistics-service" | "masterdata-service" | "identity-service" |
| 50 | +_event_type : string // e.g. "GoodsReceivedV1" |
| 51 | +_event_id : string // Wolverine's envelope id — the S3 key's dedup key |
| 52 | +_occurred_at : string // ISO-8601 — when the event happened, not when it was exported; partitions by business date |
| 53 | +``` |
| 54 | + |
| 55 | +The CDC path has no equivalent envelope of its own — `source.schema`/`source.table`/`source.lsn`/ |
| 56 | +`ts_ms` inside Debezium's native JSON serve the same role. |
| 57 | + |
| 58 | +## 3. Event-driven catalog (`bronze/`) — every field, as actually implemented |
| 59 | + |
| 60 | +One row per published integration event that has a BiExport consumer today. Field names here are the |
| 61 | +literal Parquet column names (payload keys), not the aspirational Bronze table shapes in |
| 62 | +`BI_Plan.md` §5 — those two mostly agree, but this table is the wire truth. |
| 63 | + |
| 64 | +### `warehouse-service` (queue `warehousing.bi-export`, exchange `inventory`) |
| 65 | + |
| 66 | +| `_event_type` | Payload columns | |
| 67 | +|---|---| |
| 68 | +| `GoodsReceivedV1` | `delivery_id`, `warehouse_code`, `buffer_location`, `line_count` | |
| 69 | +| `PutAwayCompletedV1` | `delivery_id` | |
| 70 | +| `StockReservedV1` | `order_id`, `fully` | |
| 71 | +| `PicksPlannedV1` | `order_id`, `pick_count`, `picks_json` (nested array as a JSON string — explode in Silver) | |
| 72 | + |
| 73 | +### `logistics-service` (queue `logistics.bi-export`, exchange `logistics`) |
| 74 | + |
| 75 | +| `_event_type` | Payload columns | |
| 76 | +|---|---| |
| 77 | +| `GoodsReceiptConfirmedV1` | `delivery_id`, `warehouse_code`, `line_count`, `lines_json` (nested — explode in Silver) | |
| 78 | +| `DeliveryAnnouncedV1` | `delivery_id`, `warehouse_code`, `supplier_party_id`, `planned_at` | |
| 79 | +| `DockSlotAssignedV1` | `delivery_id`, `warehouse_code`, `dock_code`, `dock_slot_at`, `dock_slot_to` | |
| 80 | +| `DeliveryArrivedV1` | `delivery_id`, `warehouse_code`, `arrived_at` | |
| 81 | +| `OutboundOrderPlacedV1` | `order_id`, `customer_role_id`, `warehouse_code`, `line_count`, `lines_json` (nested) | |
| 82 | +| `OutboundOrderCancelledV1` | `order_id` | |
| 83 | +| `PickingReleasedV1` | `order_id`, `warehouse_code` | |
| 84 | +| `PickConfirmedV1` | `order_id`, `sequence`, `location`, `sku`, `batch_number`, `quantity`, `unit` | |
| 85 | +| `ShipmentDispatchedV1` | `order_id`, `shipment_id`, `warehouse_code`, `carrier_role_id`, `tracking_number` (nullable) | |
| 86 | + |
| 87 | +`DeliveryAnnouncedV1`/`DockSlotAssignedV1`/`DeliveryArrivedV1` together give DataFoundry the full |
| 88 | +inbound-delivery timeline (`announced_at`/`dock_code`/`dock_slot_at`/`arrived_at`) that |
| 89 | +`GoodsReceiptConfirmedV1` alone didn't cover — join all four on `delivery_id` in Silver to build one |
| 90 | +`fact_inbound_delivery` row per delivery. Note: `supplier_party_id`/`customer_role_id`/ |
| 91 | +`carrier_role_id` are **opaque `PartyRoleRef` strings today** (dev-seed free-text labels, not a real |
| 92 | +Partner id/code) — resolving them to a Partner dimension needs the party-role crosswalk work, not yet |
| 93 | +built. |
| 94 | + |
| 95 | +### `masterdata-service` (queue `masterdata.bi-export`, exchange `catalog`) |
| 96 | + |
| 97 | +| `_event_type` | Payload columns | |
| 98 | +|---|---| |
| 99 | +| `ProductDefinedV2` | `sku`, `name`, `base_unit`, `requires_cold_chain`, `is_hazardous`, `is_batch_tracked`, `unit_weight_kg`, `unit_volume_m3`, `min_celsius` (nullable), `max_celsius` (nullable) | |
| 100 | + |
| 101 | +### Identity (`bronze/identity-service/`) |
| 102 | + |
| 103 | +| `_event_type` | Payload columns | |
| 104 | +|---|---| |
| 105 | +| `UserSnapshot` | `user_id` (Keycloak `username`, the badge number), `display_name` (nullable), `role` (comma-joined if multiple, nullable), `home_warehouse_code` (nullable), `active` | |
| 106 | + |
| 107 | +## 4. CDC catalog (`bronze-cdc/`) — tables configured, per Debezium source instance |
| 108 | + |
| 109 | +CDC exists for internal state that **never crosses a service boundary as an integration event** — |
| 110 | +adding an event purely to feed BI would couple domain code to an analytics concern it has no other |
| 111 | +reason to know about (see `BI_Plan.md` §11 for the full rationale). One Debezium Server instance per |
| 112 | +source database, each with its own `table.include.list`; DataFoundry gets the raw row image, not an |
| 113 | +intent-carrying fact, so dedup/point-in-time logic belongs entirely in Silver. |
| 114 | + |
| 115 | +| Source DB | Tables | Status | |
| 116 | +|---|---|---| |
| 117 | +| `warehouse` | `inventory.stock_items` | **Verified** — a real Postgres `INSERT` flowed through a real Debezium container to S3 in a local smoke test | |
| 118 | +| `warehouse` | `inventory.stock_movements`, `stock_reservations`, `batches`, `stocktakes`, `stocktake_count_lines`, `allocations`, `handling_units`, `handling_unit_lines` | Configured, same instance/config as the verified table — not individually run | |
| 119 | +| `warehouse` | `topology.warehouses`, `rooms`, `locations`, `docks` | Configured, not run | |
| 120 | +| `logistics` | `inbound_deliveries`, `inbound_delivery_lines`, `outbound_orders`, `outbound_order_lines`, `pick_lists`, `pick_tasks`, `shipments`, `packages` | Configured only — this Debezium instance has never been started, even in the smoke test | |
| 121 | +| `masterdata` | `partners.parties`, `party_roles`, `customer_shipping_addresses`, `catalog.product_types` | Configured only — same, never run | |
| 122 | + |
| 123 | +`catalog.product_unit_conversions` is deliberately **not** in scope — nobody has asked for it. |
| 124 | + |
| 125 | +**Important overlap to know about**: `logistics.inbound_deliveries` is in the CDC list above *and* |
| 126 | +`DeliveryAnnouncedV1`/`DockSlotAssignedV1`/`DeliveryArrivedV1` (§3) cover much of the same columns via |
| 127 | +events instead. That's a deliberate, documented exception (`BI_Plan.md` §11) — those three are named |
| 128 | +business milestones, cheaper for Silver to consume as facts than re-derived from a row diff — not an |
| 129 | +oversight. Don't build a Silver transform that expects the CDC path to be the only source for those |
| 130 | +columns. |
| 131 | + |
| 132 | +## 5. What's deliberately not here yet |
| 133 | + |
| 134 | +- **ASN accuracy** (`expected_qty`/`received_qty`/`discrepancy_qty` per delivery line) — the delivery |
| 135 | + *timeline* (§3) is covered; the line-level expected-vs-actual comparison from UC-02 is a separate, |
| 136 | + not-yet-built slice. |
| 137 | +- **Cost/valuation** (`stock_value`, `Money` anywhere) — no source system carries cost data yet; a |
| 138 | + future payments service is the anticipated origin (`BI_Plan.md` §9). |
| 139 | +- **Delivery/shipment confirmation past `Dispatched`** — out of scope for this domain today; proposed |
| 140 | + as a separate future `fleet-telemetry-service` (`BI_Plan.md` §10). |
| 141 | +- **Party-role → Partner resolution** — `PartyRoleRef` fields travel as opaque strings; a crosswalk |
| 142 | + resolving them to real Partner attributes is a real backlog item, blocked on real Partner |
| 143 | + ids/codes reaching the producing services' seed/prod data. |
| 144 | +- **No Databricks-side objects exist yet** — no workspace, no Auto Loader job, no Bronze/Silver/Gold |
| 145 | + Delta tables. Everything above describes what the Warehouse side delivers into S3; wiring |
| 146 | + Databricks to actually read it is DataFoundry's next step, not built from this repo. |
| 147 | + |
| 148 | +## 6. Versioning contract |
| 149 | + |
| 150 | +Every event name is suffixed `V<n>` (e.g. `GoodsReceiptConfirmedV1`). Wire shapes are **additive-only |
| 151 | +and immutable**: a new field ships as a new `V<n+1>` event, never as an edit to an existing one, and |
| 152 | +old and new versions can be in flight at the same time. DataFoundry's Silver mappings should key on |
| 153 | +`_event_type` (the exact versioned name), not assume a field list is stable across versions of "the |
| 154 | +same" event. |
0 commit comments