Skip to content

feat(sources): B2.2 — keyset merge-join - #73

Merged
thiagoluga merged 2 commits into
masterfrom
feat/b2-2-merge-join
Jul 1, 2026
Merged

feat(sources): B2.2 — keyset merge-join#73
thiagoluga merged 2 commits into
masterfrom
feat/b2-2-merge-join

Conversation

@thiagoluga

Copy link
Copy Markdown
Owner

The second multi-source strategy — the keyset merge-join.

Join.MergeJoin merges two sources ordered by the same key into an IStreamingSource<TResult>. For each left row it emits the contiguous group of right rows sharing its key. Inner drops unmatched left rows; LeftOuter keeps them with an empty group.

.From(Join.MergeJoin(
    left:     Source.Sql(conn, sqlCustomers).Keyset<Customer, long>(c => c.Id),
    keyLeft:  c => c.Id,
    right:    Source.Sql(conn, sqlOrders).Keyset<Order, long>(o => o.CustomerId),
    keyRight: o => o.CustomerId,
    map:      (c, orders) => new CustomerOrders(c, orders),
    kind:     JoinKind.LeftOuter))

Design

  • Streams the merge, buffering one right key-group at a time → constant memory as long as a single key's right multiplicity is bounded (both sources must be ordered by their key, same as the v1 keyset rule). The pipeline slices the stream into batches.
  • DelegatingStreamingSource<TResult> keeps the class type params low; the four-type ergonomic API lives on Join.MergeJoin (S2436 suppressed with a justification — a join needs 4 types, exactly like BCL Enumerable.GroupJoin).
  • An internal Paginate<T> reads a batch source page by page as a flat async sequence.

Tests (4 green Join total)

  • Inner emits only matched left rows with their group — right rows spanning a page boundary are gathered in order.
  • Left-outer keeps unmatched left rows with an empty group.

Epic B2 status

B2.1 (enrichment) ✅ + B2.2 (merge-join) ✅ — both strategies done. Remaining: B2.3 (packaging + sample, needs your Pro-vs-free call) and B2.4 (dynamic config, optional).

The second multi-source strategy. Join.MergeJoin merges two sources ordered by the same
key into an IStreamingSource<TResult>: for each left row it emits the contiguous group of
right rows sharing its key. Inner drops unmatched left rows; LeftOuter keeps them with an
empty group. Streams the merge (one right key-group buffered at a time) so memory is
constant when per-key multiplicity is bounded; the pipeline slices the stream into batches.

- DelegatingStreamingSource<TResult> (thin IStreamingSource over a produce delegate) keeps
  the class type params low; the four-type ergonomic API lives on Join.MergeJoin, where
  S2436 is suppressed with a justification (a join needs 4 types, like BCL GroupJoin).
- An internal Paginate<T> reads a batch source page by page as a flat async sequence.

Tests (4 green Join total): inner emits only matched left rows with their group (right
rows spanning a page boundary, in order); left-outer keeps unmatched left rows with an
empty group. PLAN B2.2 done.
@sonarqubecloud

sonarqubecloud Bot commented Jul 1, 2026

Copy link
Copy Markdown

@thiagoluga
thiagoluga merged commit 5a11e44 into master Jul 1, 2026
5 checks passed
@thiagoluga
thiagoluga deleted the feat/b2-2-merge-join branch July 1, 2026 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant