Skip to content

Commit 1a2ea3c

Browse files
feat(bi-export): land the inbound delivery timeline in Bronze (#70)
Adds DeliveryAnnouncedV1, DockSlotAssignedV1, DeliveryArrivedV1 integration events for the InboundDelivery transitions (announce/dock-slot/arrival) that previously never crossed a service boundary as an event. Publishes each through the transactional outbox from the corresponding handler (AnnounceDeliveryHandler, AssignDockSlotHandler, RegisterArrivalHandler), same pattern as the existing ConfirmReceiptHandler → GoodsReceiptConfirmedV1. Adds a sink-only BiExport consumer per event (own logistics.bi-export queue, no domain writes), landing Bronze columns for BronzeInboundDelivery.announced_at / dock_code / dock_slot_at / arrived_at (docs/BI_Plan.md §5.3). This is item #1a ("delivery timeline export") from the domain-docs gap analysis — expected/received qty (ASN accuracy) was intentionally left out of this slice.
1 parent cb52bb3 commit 1a2ea3c

15 files changed

Lines changed: 469 additions & 19 deletions

File tree

docs/warehouse_datafoundry.md

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
# Warehouse ↔ DataFoundry integration contract
2+
3+
> This is the producer-side contract for **DataFoundry** (the Databricks lakehouse project that
4+
> consumes this data) — what actually lands in S3 today, in what shape, at what path, with what
5+
> guarantees. For the *why* behind each design choice (cloud-agnostic CDC, Auto Loader vs.
6+
> alternatives, Silver/Gold design, the metrics catalog) see [`BI_Plan.md`](BI_Plan.md) — this file
7+
> only states what DataFoundry can rely on right now.
8+
>
9+
> Nothing below has been deployed to a live AWS environment yet (`BI_LAKE_BUCKET` has never been set
10+
> on a real `terraform apply`) — every guarantee here has been verified locally against real
11+
> Postgres/Debezium/RabbitMQ/S3-compatible (LocalStack) containers, not the real thing. Status per
12+
> row is called out explicitly; nothing is glossed over as "done" that hasn't actually run.
13+
14+
## 1. The landing zone
15+
16+
One S3 bucket, on/off with a single Terraform variable:
17+
18+
- **Bucket**: `warehouse-bi-lake-<account-id>` (override via `bi_lake_bucket_name`,
19+
`infra/terraform/bootstrap/variables.tf`)
20+
- **Switch**: setting the `BI_LAKE_BUCKET` repo variable turns on *both* producer paths below at
21+
once (`docs/deploy.md`) — there's no way to enable one without the other today.
22+
- **IAM**: a single shared ECS task role can `s3:PutObject` under this bucket's root; nothing else
23+
writes to it.
24+
25+
Three independent writers land into the same bucket, under three separate prefixes. DataFoundry's
26+
Auto Loader jobs should point at prefixes, not the bucket root.
27+
28+
| Prefix | Writer | Format | Trigger |
29+
|---|---|---|---|
30+
| `bronze/<source_system>/<event_type>/date=YYYY-MM-DD/<event_id>.parquet` | `S3ParquetBronzeExportWriter` (`Warehouse.ServiceDefaults.BiExport`) — one instance per backend service | Parquet, one row per file | Each service's own published integration events, via a dedicated `<service>.bi-export` RabbitMQ queue |
31+
| `bronze-cdc/<schema>/<table>/date=YYYY-MM-DD/<lsn>.json` | `Warehouse.CdcBridge` (one shared worker) | Raw Debezium change-event JSON (`before`/`after`/`op`/`source`), byte-for-byte, schemas stripped | Postgres logical replication → Debezium Server → RabbitMQ → CdcBridge, for internal state that never becomes an integration event |
32+
| `bronze/identity-service/UserSnapshot/date=YYYY-MM-DD/<event_id>.parquet` | `Warehouse.IdentityBridge` (reuses the same Parquet writer) | Parquet, one row per user per poll | Periodic poll of Keycloak's Admin REST API (no CDC/events possible — Keycloak dev-mode has no persistent DB) |
33+
34+
**Idempotency**: every key is deterministic — the event-driven path keys on the Wolverine outbox
35+
message id, CDC keys on the Postgres LSN (the WAL position of the change). A redelivery from the
36+
at-least-once outbox, or from Debezium, overwrites the same S3 object rather than producing a
37+
duplicate row. DataFoundry's ingestion does **not** need its own dedup step for either path.
38+
39+
**Schema-on-read**: every Parquet column (`bronze/...`) is written as a string, including numbers and
40+
timestamps (ISO-8601, `"O"` format) — typing happens in Silver, not Bronze, by design. The CDC path
41+
is native JSON with no imposed schema at all; Debezium's own envelope carries the typed `before`/
42+
`after` row images.
43+
44+
## 2. Common envelope (event-driven path only)
45+
46+
Every `bronze/...` Parquet file carries these columns before the event's own fields:
47+
48+
```
49+
_source_system : string // "warehouse-service" | "logistics-service" | "masterdata-service" | "identity-service"
50+
_event_type : string // e.g. "GoodsReceivedV1"
51+
_event_id : string // Wolverine's envelope id — the S3 key's dedup key
52+
_occurred_at : string // ISO-8601 — when the event happened, not when it was exported; partitions by business date
53+
```
54+
55+
The CDC path has no equivalent envelope of its own — `source.schema`/`source.table`/`source.lsn`/
56+
`ts_ms` inside Debezium's native JSON serve the same role.
57+
58+
## 3. Event-driven catalog (`bronze/`) — every field, as actually implemented
59+
60+
One row per published integration event that has a BiExport consumer today. Field names here are the
61+
literal Parquet column names (payload keys), not the aspirational Bronze table shapes in
62+
`BI_Plan.md` §5 — those two mostly agree, but this table is the wire truth.
63+
64+
### `warehouse-service` (queue `warehousing.bi-export`, exchange `inventory`)
65+
66+
| `_event_type` | Payload columns |
67+
|---|---|
68+
| `GoodsReceivedV1` | `delivery_id`, `warehouse_code`, `buffer_location`, `line_count` |
69+
| `PutAwayCompletedV1` | `delivery_id` |
70+
| `StockReservedV1` | `order_id`, `fully` |
71+
| `PicksPlannedV1` | `order_id`, `pick_count`, `picks_json` (nested array as a JSON string — explode in Silver) |
72+
73+
### `logistics-service` (queue `logistics.bi-export`, exchange `logistics`)
74+
75+
| `_event_type` | Payload columns |
76+
|---|---|
77+
| `GoodsReceiptConfirmedV1` | `delivery_id`, `warehouse_code`, `line_count`, `lines_json` (nested — explode in Silver) |
78+
| `DeliveryAnnouncedV1` | `delivery_id`, `warehouse_code`, `supplier_party_id`, `planned_at` |
79+
| `DockSlotAssignedV1` | `delivery_id`, `warehouse_code`, `dock_code`, `dock_slot_at`, `dock_slot_to` |
80+
| `DeliveryArrivedV1` | `delivery_id`, `warehouse_code`, `arrived_at` |
81+
| `OutboundOrderPlacedV1` | `order_id`, `customer_role_id`, `warehouse_code`, `line_count`, `lines_json` (nested) |
82+
| `OutboundOrderCancelledV1` | `order_id` |
83+
| `PickingReleasedV1` | `order_id`, `warehouse_code` |
84+
| `PickConfirmedV1` | `order_id`, `sequence`, `location`, `sku`, `batch_number`, `quantity`, `unit` |
85+
| `ShipmentDispatchedV1` | `order_id`, `shipment_id`, `warehouse_code`, `carrier_role_id`, `tracking_number` (nullable) |
86+
87+
`DeliveryAnnouncedV1`/`DockSlotAssignedV1`/`DeliveryArrivedV1` together give DataFoundry the full
88+
inbound-delivery timeline (`announced_at`/`dock_code`/`dock_slot_at`/`arrived_at`) that
89+
`GoodsReceiptConfirmedV1` alone didn't cover — join all four on `delivery_id` in Silver to build one
90+
`fact_inbound_delivery` row per delivery. Note: `supplier_party_id`/`customer_role_id`/
91+
`carrier_role_id` are **opaque `PartyRoleRef` strings today** (dev-seed free-text labels, not a real
92+
Partner id/code) — resolving them to a Partner dimension needs the party-role crosswalk work, not yet
93+
built.
94+
95+
### `masterdata-service` (queue `masterdata.bi-export`, exchange `catalog`)
96+
97+
| `_event_type` | Payload columns |
98+
|---|---|
99+
| `ProductDefinedV2` | `sku`, `name`, `base_unit`, `requires_cold_chain`, `is_hazardous`, `is_batch_tracked`, `unit_weight_kg`, `unit_volume_m3`, `min_celsius` (nullable), `max_celsius` (nullable) |
100+
101+
### Identity (`bronze/identity-service/`)
102+
103+
| `_event_type` | Payload columns |
104+
|---|---|
105+
| `UserSnapshot` | `user_id` (Keycloak `username`, the badge number), `display_name` (nullable), `role` (comma-joined if multiple, nullable), `home_warehouse_code` (nullable), `active` |
106+
107+
## 4. CDC catalog (`bronze-cdc/`) — tables configured, per Debezium source instance
108+
109+
CDC exists for internal state that **never crosses a service boundary as an integration event**
110+
adding an event purely to feed BI would couple domain code to an analytics concern it has no other
111+
reason to know about (see `BI_Plan.md` §11 for the full rationale). One Debezium Server instance per
112+
source database, each with its own `table.include.list`; DataFoundry gets the raw row image, not an
113+
intent-carrying fact, so dedup/point-in-time logic belongs entirely in Silver.
114+
115+
| Source DB | Tables | Status |
116+
|---|---|---|
117+
| `warehouse` | `inventory.stock_items` | **Verified** — a real Postgres `INSERT` flowed through a real Debezium container to S3 in a local smoke test |
118+
| `warehouse` | `inventory.stock_movements`, `stock_reservations`, `batches`, `stocktakes`, `stocktake_count_lines`, `allocations`, `handling_units`, `handling_unit_lines` | Configured, same instance/config as the verified table — not individually run |
119+
| `warehouse` | `topology.warehouses`, `rooms`, `locations`, `docks` | Configured, not run |
120+
| `logistics` | `inbound_deliveries`, `inbound_delivery_lines`, `outbound_orders`, `outbound_order_lines`, `pick_lists`, `pick_tasks`, `shipments`, `packages` | Configured only — this Debezium instance has never been started, even in the smoke test |
121+
| `masterdata` | `partners.parties`, `party_roles`, `customer_shipping_addresses`, `catalog.product_types` | Configured only — same, never run |
122+
123+
`catalog.product_unit_conversions` is deliberately **not** in scope — nobody has asked for it.
124+
125+
**Important overlap to know about**: `logistics.inbound_deliveries` is in the CDC list above *and*
126+
`DeliveryAnnouncedV1`/`DockSlotAssignedV1`/`DeliveryArrivedV1` (§3) cover much of the same columns via
127+
events instead. That's a deliberate, documented exception (`BI_Plan.md` §11) — those three are named
128+
business milestones, cheaper for Silver to consume as facts than re-derived from a row diff — not an
129+
oversight. Don't build a Silver transform that expects the CDC path to be the only source for those
130+
columns.
131+
132+
## 5. What's deliberately not here yet
133+
134+
- **ASN accuracy** (`expected_qty`/`received_qty`/`discrepancy_qty` per delivery line) — the delivery
135+
*timeline* (§3) is covered; the line-level expected-vs-actual comparison from UC-02 is a separate,
136+
not-yet-built slice.
137+
- **Cost/valuation** (`stock_value`, `Money` anywhere) — no source system carries cost data yet; a
138+
future payments service is the anticipated origin (`BI_Plan.md` §9).
139+
- **Delivery/shipment confirmation past `Dispatched`** — out of scope for this domain today; proposed
140+
as a separate future `fleet-telemetry-service` (`BI_Plan.md` §10).
141+
- **Party-role → Partner resolution**`PartyRoleRef` fields travel as opaque strings; a crosswalk
142+
resolving them to real Partner attributes is a real backlog item, blocked on real Partner
143+
ids/codes reaching the producing services' seed/prod data.
144+
- **No Databricks-side objects exist yet** — no workspace, no Auto Loader job, no Bronze/Silver/Gold
145+
Delta tables. Everything above describes what the Warehouse side delivers into S3; wiring
146+
Databricks to actually read it is DataFoundry's next step, not built from this repo.
147+
148+
## 6. Versioning contract
149+
150+
Every event name is suffixed `V<n>` (e.g. `GoodsReceiptConfirmedV1`). Wire shapes are **additive-only
151+
and immutable**: a new field ships as a new `V<n+1>` event, never as an edit to an existing one, and
152+
old and new versions can be in flight at the same time. DataFoundry's Silver mappings should key on
153+
`_event_type` (the exact versioned name), not assume a field list is stable across versions of "the
154+
same" event.
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
namespace Warehouse.Contracts.Logistics;
2+
3+
/// <summary>
4+
/// Integration event: an inbound delivery (ASN) was announced (UC-01). Published by Logistics
5+
/// through the transactional outbox right after <c>InboundDelivery.Announce</c>; feeds the delivery
6+
/// timeline in Bronze (docs/BI_Plan.md §5.3 — <c>BronzeInboundDelivery.announced_at</c>).
7+
///
8+
/// <b>Additive-only.</b> This is the V1 wire shape and must never change — a new field ships as
9+
/// <c>DeliveryAnnouncedV2</c>, never as an edit here. Primitives only.
10+
/// </summary>
11+
public sealed record DeliveryAnnouncedV1(
12+
Guid DeliveryId,
13+
string WarehouseCode,
14+
string SupplierRoleId,
15+
DateTimeOffset PlannedAt,
16+
DateTimeOffset OccurredAt);
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
namespace Warehouse.Contracts.Logistics;
2+
3+
/// <summary>
4+
/// Integration event: the truck checked in at the dock for an announced delivery (UC-02). Published
5+
/// by Logistics through the transactional outbox right after <c>InboundDelivery.RegisterArrival</c>;
6+
/// feeds the delivery timeline in Bronze (docs/BI_Plan.md §5.3 —
7+
/// <c>BronzeInboundDelivery.arrived_at</c>).
8+
///
9+
/// <b>Additive-only.</b> This is the V1 wire shape and must never change — a new field ships as
10+
/// <c>DeliveryArrivedV2</c>, never as an edit here. Primitives only.
11+
/// </summary>
12+
public sealed record DeliveryArrivedV1(
13+
Guid DeliveryId,
14+
string WarehouseCode,
15+
DateTimeOffset OccurredAt);
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
namespace Warehouse.Contracts.Logistics;
2+
3+
/// <summary>
4+
/// Integration event: a dock + time window was booked for an announced delivery (UC-01). Published
5+
/// by Logistics through the transactional outbox right after <c>InboundDelivery.AssignDockSlot</c>;
6+
/// feeds the delivery timeline in Bronze (docs/BI_Plan.md §5.3 —
7+
/// <c>BronzeInboundDelivery.dock_code</c>/<c>dock_slot_at</c>).
8+
///
9+
/// <b>Additive-only.</b> This is the V1 wire shape and must never change — a new field ships as
10+
/// <c>DockSlotAssignedV2</c>, never as an edit here. Primitives only.
11+
/// </summary>
12+
public sealed record DockSlotAssignedV1(
13+
Guid DeliveryId,
14+
string WarehouseCode,
15+
string DockCode,
16+
DateTimeOffset SlotFrom,
17+
DateTimeOffset SlotTo,
18+
DateTimeOffset OccurredAt);
Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
using Warehouse.Contracts.Logistics;
2+
using Warehouse.ServiceDefaults.BiExport;
3+
using Wolverine;
4+
5+
namespace Warehouse.Logistics.Core.Application.BiExport;
6+
7+
/// <summary>
8+
/// Exports a copy of every <see cref="DeliveryAnnouncedV1"/> to the Bronze landing zone
9+
/// (docs/BI_Plan.md §5.3/§11 — feeds <c>BronzeInboundDelivery.announced_at</c>). Same shape as
10+
/// <c>GoodsReceiptConfirmedBiExportConsumer</c>: a pure side observer on its own dedicated queue, no
11+
/// domain writes, no outbox.
12+
/// </summary>
13+
public sealed class DeliveryAnnouncedBiExportConsumer(IBronzeExportWriter writer)
14+
{
15+
private const string SourceSystem = "logistics-service";
16+
17+
public Task Handle(DeliveryAnnouncedV1 message, Envelope envelope, CancellationToken cancellationToken)
18+
{
19+
var bronzeEvent = new BronzeEvent(
20+
SourceSystem,
21+
EventType: nameof(DeliveryAnnouncedV1),
22+
EventId: envelope.Id.ToString(),
23+
message.OccurredAt,
24+
Payload: new Dictionary<string, string?>
25+
{
26+
["delivery_id"] = message.DeliveryId.ToString(),
27+
["warehouse_code"] = message.WarehouseCode,
28+
["supplier_party_id"] = message.SupplierRoleId,
29+
["planned_at"] = message.PlannedAt.ToString("O"),
30+
});
31+
32+
return writer.WriteAsync(bronzeEvent, cancellationToken);
33+
}
34+
}
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
using Warehouse.Contracts.Logistics;
2+
using Warehouse.ServiceDefaults.BiExport;
3+
using Wolverine;
4+
5+
namespace Warehouse.Logistics.Core.Application.BiExport;
6+
7+
/// <summary>
8+
/// Exports a copy of every <see cref="DeliveryArrivedV1"/> to the Bronze landing zone
9+
/// (docs/BI_Plan.md §5.3/§11 — feeds <c>BronzeInboundDelivery.arrived_at</c>). Same shape as
10+
/// <c>GoodsReceiptConfirmedBiExportConsumer</c>: a pure side observer on its own dedicated queue, no
11+
/// domain writes, no outbox.
12+
/// </summary>
13+
public sealed class DeliveryArrivedBiExportConsumer(IBronzeExportWriter writer)
14+
{
15+
private const string SourceSystem = "logistics-service";
16+
17+
public Task Handle(DeliveryArrivedV1 message, Envelope envelope, CancellationToken cancellationToken)
18+
{
19+
var bronzeEvent = new BronzeEvent(
20+
SourceSystem,
21+
EventType: nameof(DeliveryArrivedV1),
22+
EventId: envelope.Id.ToString(),
23+
message.OccurredAt,
24+
Payload: new Dictionary<string, string?>
25+
{
26+
["delivery_id"] = message.DeliveryId.ToString(),
27+
["warehouse_code"] = message.WarehouseCode,
28+
["arrived_at"] = message.OccurredAt.ToString("O"),
29+
});
30+
31+
return writer.WriteAsync(bronzeEvent, cancellationToken);
32+
}
33+
}
Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
using Warehouse.Contracts.Logistics;
2+
using Warehouse.ServiceDefaults.BiExport;
3+
using Wolverine;
4+
5+
namespace Warehouse.Logistics.Core.Application.BiExport;
6+
7+
/// <summary>
8+
/// Exports a copy of every <see cref="DockSlotAssignedV1"/> to the Bronze landing zone
9+
/// (docs/BI_Plan.md §5.3/§11 — feeds <c>BronzeInboundDelivery.dock_code</c>/<c>dock_slot_at</c>).
10+
/// Same shape as <c>GoodsReceiptConfirmedBiExportConsumer</c>: a pure side observer on its own
11+
/// dedicated queue, no domain writes, no outbox.
12+
/// </summary>
13+
public sealed class DockSlotAssignedBiExportConsumer(IBronzeExportWriter writer)
14+
{
15+
private const string SourceSystem = "logistics-service";
16+
17+
public Task Handle(DockSlotAssignedV1 message, Envelope envelope, CancellationToken cancellationToken)
18+
{
19+
var bronzeEvent = new BronzeEvent(
20+
SourceSystem,
21+
EventType: nameof(DockSlotAssignedV1),
22+
EventId: envelope.Id.ToString(),
23+
message.OccurredAt,
24+
Payload: new Dictionary<string, string?>
25+
{
26+
["delivery_id"] = message.DeliveryId.ToString(),
27+
["warehouse_code"] = message.WarehouseCode,
28+
["dock_code"] = message.DockCode,
29+
["dock_slot_at"] = message.SlotFrom.ToString("O"),
30+
["dock_slot_to"] = message.SlotTo.ToString("O"),
31+
});
32+
33+
return writer.WriteAsync(bronzeEvent, cancellationToken);
34+
}
35+
}

src/Services/Logistics/Modules/Warehouse.Logistics.Core/Application/Deliveries/AnnounceDelivery/AnnounceDeliveryHandler.cs

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,23 @@
1+
using Warehouse.Contracts.Logistics;
12
using Warehouse.Logistics.Core.Application.Abstractions;
23
using Warehouse.Logistics.Core.Domain;
3-
using Warehouse.SharedKernel.Application;
4+
using Warehouse.Logistics.Core.Infrastructure.Persistence;
45
using Warehouse.SharedKernel.Domain;
56
using Warehouse.SharedKernel.ValueObjects;
7+
using Wolverine.EntityFrameworkCore;
68

79
namespace Warehouse.Logistics.Core.Application.Deliveries.AnnounceDelivery;
810

911
/// <summary>
1012
/// Validates every announced SKU against the local Catalog replica (ADR-0003 — no cross-service
1113
/// query) and creates the <see cref="InboundDelivery"/>. An ASN that names an unknown SKU is
1214
/// rejected so it never enters the receiving flow with a code Inventory cannot resolve.
15+
/// Announces the ASN to Bronze via the transactional outbox (<see cref="DeliveryAnnouncedV1"/>).
1316
/// </summary>
1417
public sealed class AnnounceDeliveryHandler(
1518
IInboundDeliveryRepository deliveries,
1619
ICatalogProductReplica catalog,
17-
IUnitOfWork unitOfWork)
20+
IDbContextOutbox<LogisticsDbContext> outbox)
1821
{
1922
public async Task<Guid> HandleAsync(AnnounceDeliveryCommand command, CancellationToken cancellationToken = default)
2023
{
@@ -52,7 +55,10 @@ public async Task<Guid> HandleAsync(AnnounceDeliveryCommand command, Cancellatio
5255
lines);
5356

5457
deliveries.Add(delivery);
55-
await unitOfWork.SaveChangesAsync(cancellationToken);
58+
await outbox.PublishAsync(new DeliveryAnnouncedV1(
59+
delivery.Id.Value, delivery.Warehouse.Code, delivery.Supplier.Value, delivery.PlannedAt, DateTimeOffset.UtcNow));
60+
61+
await outbox.SaveChangesAndFlushMessagesAsync(cancellationToken);
5662
return delivery.Id.Value;
5763
}
5864
}

0 commit comments

Comments
 (0)