Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions DECISIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,8 @@ Removed from v1 vs. Ch. 16: `IRetryPolicy`, `IExceptionClassifier`, `IAuthProvid
| D21 | Dynamic path | Reopened in v2. Row type = positional `ReportRecord` (`object?[]` + `ReportSchema`), not a dictionary; reuses the whole v1 pipeline because the writer edge already speaks `object?[]` + schema. Config (JSON) + `IReportConfigParser` + JsonLogic filter return additively to `Abstractions` (SemVer-minor) |
| D22 | Multi-sheet XLSX | **First v2 paid feature (maintainer-locked).** One workbook, several named sheets, each from a different filter over the same source (different sources per sheet = B2). Single pass preserved via a generic "multi-section output" hook in OSS Core; the XLSX workbook writer + fluent API ship in the commercial `NeoReports.Xlsx.Pro` package (D27). Blueprint: `docs/epic-b1-multisheet-pro.md`. Some sub-decisions still open there |
| D27 | Pro package model | **Open-core** (forced by the already-MIT core). Core stays MIT; advanced features ship in **`NeoReports.Xlsx.Pro`** — **source-available, Option A (QuestPDF-style): free under USD 1M annual revenue, paid above** (use **PolyForm Small Business 1.0.0**; fetch canonical text at B1.2). **No runtime enforcement** for now (contractual, like QuestPDF). Pro plugs in via existing extensibility; the OSS core never depends on Pro. Pro packages are excluded from the OSS NuGet release; commercial sales terms are the maintainer's/lawyer's. The **multi-view hook is MIT** (one file per view); Pro adds the single-workbook (sheets) writer |
| D28 | Multi-source join (B2) | **Two explicit, user-chosen strategies** (not auto-detected): (a) **keyset merge-join** — an `IStreamingSource<TResult>` that merges two sources ordered by the same key, constant memory when per-key multiplicity is bounded, inner + left-outer; (b) **enrichment/lookup** — an `IBatchSource<TResult>` that, per page of a primary source, makes ONE batched lookup call and maps it in (O(pageSize), structurally no N+1). Both produce a source the existing pipeline consumes unchanged. Blueprint: `docs/epic-b2-multisource.md` |
| D29 | Multi-source packaging | **Open (maintainer):** B2's value is the join sources themselves, so there is no natural MIT/Pro split like B1 — it is a straight monetization call. Recommend **Pro** (`NeoReports.Sources.Join.Pro`, same model as D27) for consistency with the B1 decision; the free-MIT alternative maximizes adoption but forgoes B2 revenue. Also open: v1 join types (inner + left-outer proposed), dynamic-config support (defer), and the user-validation gate |
| D23 | Multi-source | Planned for v2 (Epic B2). Any report assembled from several sources (join/enrich). Likely Pro. **Two explicit, user-chosen strategies** (not auto-detected): keyset **merge-join** of two ordered sources (constant memory) and per-row **enrichment/lookup** (batched per page). Reuses the workbook writer for multi-source-per-sheet. Design recorded before coding |
| D24 | UI ordering | Blazor UI is the **last** v2 epic (after dynamic path + multi-source + a user-validation gate), per the maintainer. Always built from the Claude Design handoff, never invented |
| D25 | v2 additivity | Every v2 addition is additive and SemVer-minor on `Abstractions`; v1's frozen surface is never broken, only extended. Removing anything stays SemVer-major |
Expand Down
19 changes: 15 additions & 4 deletions PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,10 +156,21 @@ boundary) are still open in that doc and must be settled before B1.2.
resolves an `ISectionedWriterFactory` by format and builds `ToSections(...)`. `AddXlsxWorkbook()` DI
helper registers the Pro writer (format `xlsx-workbook`). ✅ 56 green Core tests (+1) + Pro DI test.

### B2 — Multi-source reports (later)

- [ ] Any report assembled from several sources (join/enrich) into one output — likely also Pro.
Design (join semantics, memory/perf) recorded before coding. See **D23**.
### B2 — Multi-source reports (join / enrichment)

Blueprint: [`docs/epic-b2-multisource.md`](docs/epic-b2-multisource.md); **D28** (two strategies) and
**D29** (packaging/monetization — open). **Two explicit, user-chosen strategies**; both produce a
source the existing pipeline consumes unchanged. Open sub-decisions (Pro vs free, package name, join
types, dynamic config, validation gate) in the doc must be settled before B2.3.

- [ ] **B2.1 — Enrichment** (`.Enrich(key, lookup, map)`): an `IBatchSource<TResult>` wrapper that
batch-looks-up related data once per page and maps it in (O(pageSize), no N+1). Tests: batched
per page, correct mapping, missing-key handling.
- [ ] **B2.2 — Keyset merge-join** (`Source.MergeJoin(left, right, on, map)`): a streaming merge of
two same-key-ordered sources; inner + left-outer; constant memory (bounded key group). Tests:
ordered-merge correctness, memory, Testcontainers E2E across two SQL sources.
- [ ] **B2.3 — Package & docs + sample** `07-multi-source` (per the Pro/free decision).
- [ ] **B2.4 — Dynamic config** for multi-source (optional, later).

### Backlog (cross-cutting)

Expand Down
92 changes: 92 additions & 0 deletions docs/epic-b2-multisource.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Epic B2 — Multi-source reports (join / enrichment)

> **Status: design, not yet built.** Blueprint to approve before coding. Decisions land in
> `DECISIONS.md` (D23 expanded, D28/D29 new).

## Goal

Let one report be assembled from **several sources** — combining rows by key into one output. Two
shapes, kept as **two explicit, user-chosen strategies** (D23) because they have different memory/perf
profiles and forcing one abstraction hurts:

1. **Keyset merge-join** — two sources ordered by the same key, merged in streaming. Constant memory;
best for joining two large ordered datasets.
2. **Enrichment / lookup** — for each page of a primary source, batch-fetch related data from a
secondary and map it in. O(pageSize) memory; best for "for each row, get X from another source."

Both produce an `IBatchSource<TResult>` (or `IStreamingSource<TResult>`) that the **existing pipeline
consumes unchanged** — no parallel pipeline, same batch/retry/writer/destination path.

## Strategy A — keyset merge-join

A streaming merge of two ordered sources:

```csharp
var report = new ReportBuilder<CustomerOrders>("...")
.From(Source.MergeJoin(
left: Source.Sql(conn, sqlCustomers).Keyset<Customer, long>(c => c.Id, pageSize: 1000),
right: Source.Sql(conn, sqlOrders).Keyset<Order, long>(o => o.CustomerId, pageSize: 1000),
on: (c, o) => c.Id.CompareTo(o.CustomerId),
map: (c, orders) => new CustomerOrders(c, orders))) // orders = the matched right rows
...
```

- Both sources **must be ordered by the join key** (the keyset key) — a documented precondition, same
spirit as the v1 keyset requirement.
- Implemented as an `IStreamingSource<TResult>`: page each sub-source into an ordered async stream,
advance the side with the smaller key, group equal keys, emit results. The pipeline slices the
stream into batches (D4).
- **Constant memory** as long as the multiplicity of a single key is bounded (one key-group from each
side held at the merge frontier). Document this; a pathological one-key-maps-to-everything join is
the caller's responsibility.
- Join types for v1: **inner** and **left-outer** (`map` receives an empty/absent right group).

## Strategy B — enrichment / lookup

An `IBatchSource<TResult>` wrapping a primary source:

```csharp
.From(Source.Sql(conn, sqlCustomers).Keyset<Customer, long>(c => c.Id)
.Enrich(
key: c => c.Id,
lookup: async (keys, ct) => await LoadOrderCountsAsync(keys, ct), // ONE batched call per page
map: (c, orderCount) => new CustomerSummary(c, orderCount)))
```

- Per page: read the primary page, collect its keys, make **one batched lookup call** for the whole
page (never one-per-row), then map each row + its looked-up value. Cursor = the primary's cursor.
- **O(pageSize)** memory. The batched-per-page shape structurally prevents the N+1 trap.
- The lookup is a user delegate (any source: SQL `WHERE key IN (...)`, HTTP, cache, ...).

## OSS / Pro boundary (needs a maintainer decision)

Unlike B1 (a generic MIT hook + a Pro writer), B2's value **is** the join sources themselves — there
is no natural "free generic half." So this is a straight monetization call:

- **Recommended: Pro** — ship both strategies in a commercial package (e.g. `NeoReports.Sources.Join.Pro`),
same model as `NeoReports.Xlsx.Pro` (PolyForm Small Business, `IsPackable=false`, excluded from the
OSS release). Consistent with "advanced features are paid" (D27) and the maintainer's B1 choice.
- **Alternative: free (MIT)** in `NeoReports.Sources.Join` — maximizes adoption, forgoes B2 revenue.

They plug in through the existing extensibility (`IBatchSource`/`IStreamingSource` + the fluent
`Source.MergeJoin` / `.Enrich`), so the OSS engine is unchanged either way.

## Open sub-decisions (maintainer)

1. **Pro or free**, and if Pro, the **package name** (`NeoReports.Sources.Join.Pro`?).
2. **Join types** in v1 — inner + left-outer enough? (right/full-outer later.)
3. **Dynamic config** for multi-source — express two sources + join in JSON. Recommend **deferring**
to a later step once the typed API settles (as B1.6 followed B1.3).
4. **Validation gate** — D23/the roadmap gate multi-source on real-user validation; confirm we build it now.

## Implementation PR breakdown (after approval)

- **B2.1 — Enrichment** (`.Enrich(...)`): the simpler `IBatchSource<TResult>` wrapper + batched lookup.
Tests: batched-per-page (no N+1), correct mapping, missing-key handling.
- **B2.2 — Merge-join** (`Source.MergeJoin(...)`): the streaming keyset merge; inner + left-outer.
Tests: ordered merge correctness, constant memory (bounded key group), Testcontainers E2E across two
SQL sources.
- **B2.3 — Package & docs** (per the Pro/free decision) + a sample `07-multi-source`.
- **B2.4 — Dynamic config** for multi-source (optional, later).

Each PR small, green tests, one at a time — same workflow as Epic A/B1.