Conjoined → UseTenantPartitionedEvents migration tool

[jeremydmiller]

Background

Marten 9 supports Events.UseTenantPartitionedEvents = true (introduced #4596), which physically partitions mt_events and mt_streams by tenant_id and gives each tenant its own event sequence. The feature requires TenancyStyle.Conjoined.

No tool exists to migrate an existing conjoined event store to use tenant partitioning. Customers maturing into this feature today have to write custom SQL and accept the operational risk of bespoke data movement at scale. This issue tracks the first-class migration tool.

Reverse migration (partitioned → conjoined) is deferred — only build if a customer needs it. Per-tenant sequences may diverge, complicating the reverse path; cost outweighs current benefit.

Constraints from prior analysis

• PG partitioning requires the partition key to be part of the PK. mt_events.pk shape changes from (stream_id, seq_id) to (stream_id, seq_id, tenant_id).
• Each tenant gets its own sequence: mt_events_sequence_{suffix}.
• A 20M-event single-transaction migration locks the table for hours. The tool uses the DETACH/ATTACH PARTITION pattern so per-tenant work happens in bounded windows.
• mt_event_progression row identities must be constructed via ShardName.Compose (per jasperfx/jasperfx#419 design principle — never hand-roll).

Data policy

Do not renumber historical events. Renumbering invalidates every downstream consumer that captured a sequence position (progression rows, downstream warehouses, audit logs, external integrations). The blast radius vastly outweighs the cosmetic value of contiguous per-tenant sequences.

Per-tenant sequence start value: MAX(seq_id) FROM mt_events WHERE tenant_id = $1 + 1, computed per tenant during migration. Avoids the giant sequence hole that would otherwise appear at each tenant's first post-migration event. The (stream, seq, tenant) PK makes seq-id reuse safe across tenants (different rows).

Migration shape

Three phases. The whole tool is offline-first in v1 — operators take a downtime window for the migration. Online migration (writes continue during migration via per-tenant brief-lock windows) is a follow-up if needed.

Phase 1 — Prepare

• Validate prerequisites:
  • TenancyStyle = Conjoined (the source state required for migration)
  • Events.UseTenantPartitionedEvents = true in target config
  • Every event row has a non-null tenant_id
  • mt_tenant_partitions is populated for every tenant with events (or auto-populate from SELECT DISTINCT tenant_id)
• Take inventory:
  • Tenant list + per-tenant event count + per-tenant MAX(seq_id)
  • Existing mt_event_progression row inventory (for the audit pass — see Phase 3)
  • Estimated migration time per tenant (rough, based on event count)
• Create new partitioned parent tables alongside the existing ones:
  • mt_events_new with PARTITION BY LIST (tenant_id) and PK (stream_id, seq_id, tenant_id)
  • mt_streams_new with PARTITION BY LIST (tenant_id) and the corresponding PK extension
• Output a migration plan to stdout for operator review before Phase 2 begins.

Phase 2 — Per-tenant copy + attach

For each tenant, one at a time (sequential, with operator-controllable concurrency cap for very small tenants):

1. Compute max_seq = MAX(seq_id) FROM mt_events WHERE tenant_id = $tenant
2. Create non-partitioned mt_events_tenant_{suffix} matching the partition shape
3. INSERT INTO mt_events_tenant_{suffix} SELECT ... FROM mt_events WHERE tenant_id = $tenant (or COPY if faster — TBD by implementer)
4. Same for mt_streams_tenant_{suffix}
5. Create per-tenant sequence: CREATE SEQUENCE mt_events_sequence_{suffix} START WITH (max_seq + 1)
6. ALTER TABLE mt_events_new ATTACH PARTITION mt_events_tenant_{suffix} FOR VALUES IN ($tenant)
7. Same for mt_streams_new
8. Seed the per-tenant high-water progression row:
   • INSERT INTO mt_event_progression (name, last_seq_id) VALUES (ShardName.Compose(ShardState.HighWaterMark, tenantId: $tenant).Identity, max_seq)
   • Use ON CONFLICT (name) DO UPDATE SET last_seq_id = EXCLUDED.last_seq_id to be idempotent
9. Verify partition is correctly attached and row count matches the per-tenant inventory from Phase 1
10. Log per-tenant completion + duration

Recoverability: a failed tenant migration can be retried independently. Completed tenants are tracked in a migration-state table written to mt_tenant_migration_log (or similar) so resumption picks up where it left off.

Phase 3 — Swap + cleanup

• Audit pass: walk mt_event_progression, ensure every row is either a known store-global identity ({Projection}:All, {Projection}:V{N}:All, HighWaterMark) or a per-tenant identity matching ShardName.TryParse. Flag any hand-rolled stragglers — these need operator attention before swap.
• Swap parent tables in a single short transaction:
  • BEGIN; ALTER TABLE mt_events RENAME TO mt_events_old; ALTER TABLE mt_events_new RENAME TO mt_events; COMMIT;
  • Same for mt_streams
• Keep mt_events_old and mt_streams_old around for operator-controlled grace period (default 7 days; configurable). Manual drop after verification.
• Update mt_event_progression HighWaterMark row to the global max for backward compatibility with code that reads the legacy single high-water (if any remains post-jasperfx#419).
• Print summary: tenants migrated, events moved, time per tenant, post-migration disk usage delta.

Per-tenant high-water progression rows

Critical addition from review: the migration MUST seed mt_event_progression rows for the per-tenant high-water marks. Without this:

• The HighWaterDetector would have to lazy-INSERT them on first detection, which under load is a hot-path INSERT spike.
• Operators couldn't see per-tenant catch-up state immediately post-migration.

Phase 2 step 8 above handles this. Identity construction goes through ShardName.Compose(ShardState.HighWaterMark, tenantId: t).Identity — never hand-rolled (depends on #419 / marten#4681 landing first, since those refactor the HighWaterDetector's SQL string concat).

Tool shape

CLI subcommand under the existing marten console tool (src/CommandLineRunner/):

marten migrate-to-tenant-partitioning [options]
  --connection <name>           Target database (default: configured store)
  --dry-run                     Phase 1 only: inventory + plan, no data movement
  --resume                      Skip tenants already completed (per migration-state table)
  --tenants <list>              Migrate only these tenant ids (default: all)
  --grace-period-days <N>       Days to retain old tables before manual drop (default: 7)
  --batch-size <N>              Rows per INSERT batch for large tenants (default: 100000)

Reverse migration (partitioned → conjoined) is not part of this tool. Defer until a customer asks.

Test plan

Use the Marten.ScaleTesting harness (#4666) to validate at scale:

1. Seed 20M events under conjoined tenancy across N tenants (Telehealth domain).
2. Run marten migrate-to-tenant-partitioning --dry-run. Verify inventory matches expectation.
3. Run marten migrate-to-tenant-partitioning. Verify all tenants migrated.
4. Critical: immediately append fresh event to each tenant. Assert no PK collision, no sequence collision, correct seq_id value (max_seq + 1 for that tenant).
5. Run a projection rebuild on the migrated data. Assert correct aggregate state (compare to pre-migration baseline).
6. Run continuous catch-up. Assert new events flow through correctly per tenant.
7. Drop a tenant via DeleteAllTenantDataAsync. Assert clean partition drop, no orphaned per-tenant sequence (also closes the loop on #TBD — drop-tenant gaps issue).
8. Resume test: kill the migration tool partway, restart with --resume. Verify completed tenants are skipped, in-progress tenant is retried, final state is correct.
9. Failure test: induce a failure during one tenant's migration (e.g., constraint violation on a forced bad row). Verify other tenants completed successfully, failed tenant cleanly rollback-able.

Acceptance

• marten migrate-to-tenant-partitioning CLI command exists and is documented.
• Dry-run produces a complete plan without touching data.
• Full run successfully migrates 20M events across 50 tenants in the harness.
• Post-migration: appending new events to each tenant works on the first try — no PK collisions, no sequence collisions.
• Per-tenant high-water progression rows are seeded correctly using ShardName.Compose.
• Old tables retained per --grace-period-days; can be manually dropped after operator verification.
• Migration-state table (mt_tenant_migration_log or similar) supports --resume after partial failure.
• Documentation: new section in docs/events/multitenancy.md covering the migration tool, the data policy (no renumbering), the operational checklist (downtime window, grace period, backup recommendation), and the post-migration validation steps.

Non-goals

• Not an online migration in v1. Operators take a downtime window. Online support (per-tenant brief-lock windows, dual-write during migration) is a follow-up if needed.
• Not renumbering historical events. Sequence numbers stay as they are; per-tenant sequences start at max(seq) + 1 for that tenant.
• Not reverse migration (partitioned → conjoined). Deferred until customer demand emerges.
• Not automatic schema rollback if a migration fails partway. Recovery is operator-driven: rerun with --resume, or roll back manually by dropping the new partitioned tables (the original mt_events / mt_streams are untouched until Phase 3 swap).
• Not zero-downtime cross-version migration (e.g., Marten 8 → 9 conjoined → 9 partitioned in one step). Customer should upgrade Marten first, then run this tool.

Open operational considerations (resolved during implementation)

• Disk space: during migration both old + new tables coexist. Operator needs ~2x the current event-store disk size. Document this prominently.
• Backup strategy: tool should print a "have you backed up?" prompt at Phase 2 entry unless --skip-backup-check is passed.
• Connection pool sizing: per-tenant COPY operations are short-lived but heavy. Default pool size is usually fine; document if a customer needs to bump it.

Dependencies

• jasperfx/jasperfx#419 and marten#4681 — ShardName.Compose refactor. Migration tool uses ShardName.Compose(ShardState.HighWaterMark, tenantId: t).Identity for per-tenant high-water rows; refactor must land first so the hand-roll path doesn't drift.
• Marten.ScaleTesting: composite-projection rebuild load test harness on a Telehealth-derived domain #4666 (Marten.ScaleTesting harness) — required for the test plan above.

Related

• Per-tenant partitioning #209: Marten impl (composite, partitioning, per-tenant sequences, vectorized high-water) #4596 — original UseTenantPartitionedEvents feature
• Cap concurrent projection rebuilds per database (MaxConcurrentRebuildsPerDatabase) jasperfx#420 — rebuild throttle (relevant post-migration when operators rebuild projections under the new partitioning)
• (forthcoming drop-tenant gaps issue — Item Eliminate the temporary IDocument crutch #4 in planning thread) — shares test infrastructure with this issue's drop-tenant validation

[jeremydmiller]

Phase 1 (Prepare / dry-run) landed in #4700

Started the migration tool. Two notes on how it's structured:

Engine in core, CLI as a thin wrapper. Marten's CLI commands live in the external Marten.CommandLine package (RunJasperFxCommands), not this repo, so the migration engine is a first-class MartenDatabase API (CreateTenantPartitioningMigrationPlanAsync) — directly testable and reusable — and the marten migrate-to-tenant-partitioning subcommand becomes a wrapper added to that package later.

Phase 2 drives Marten's own partition machinery. Rather than hand-rolling the partitioned mt_events/mt_streams DDL (which would drift from EventsTable/StreamsTable), the per-tenant copy+attach phase will drive ManagedListPartitions.AddPartitionToAllTables + PerTenantEventSequences.EnsureSequencesAsync — the same path AddMartenManagedTenantsAsync uses — so the new schema is byte-identical to a natively-partitioned store.

Phase 1 ships the read-only --dry-run: prerequisite validation (conjoined source, partitioning target config, no null tenant_id), per-tenant inventory (count + MAX(seq_id) for the START WITH max+1 sequence seed), and the mt_event_progression audit classifier (store-global/per-tenant high water vs shard vs hand-rolled straggler). No data is touched.

Sequenced next: Phase 2 (copy + attach + per-tenant sequence + high-water seeding via HighWaterShardIdentity.PerTenant, plus the --resume state table), Phase 3 (swap + cleanup), the CLI wrapper, docs, and the 20M-event / 50-tenant Marten.ScaleTesting validation.