feat: Streaming initial sync for new realms (paginated sync)

[liz709]

Problem

Initial sync for users with large datasets (10k–20k+ objects) transfers and applies the entire dataset in a single atomic operation. This causes:

• Memory pressure — 20–50 MB JSON response held in memory
• Long-blocking transactions — 120k+ IndexedDB writes in one transaction, blocking UI
• Poor UX — no progress feedback during what can be a multi-second operation
• Resource limit crashes — e.g. Cloudflare Durable Objects (30s CPU limit) always fail for large datasets

This is a general UX issue for any Dexie Cloud user with a large initial dataset, not specific to any deployment environment.

Proposed Solution

Stream the initial download of new realms as NDJSON (outside the main ACID transaction), similar to how Y.js data is already handled via registerYDownloadEndpoint.ts.

Why new realms only (for now)?

• No client changes to apply → no conflict resolution needed
• Much simpler than paginating ongoing delta sync
• Covers the primary pain point (cold start / first sync)

Consistency model (3 steps)

1. Normal sync for existing realms → rev1
2. Stream new realm(s) with read-isolated PG transaction → rev2
3. If rev2 > rev1: re-run normal sync, but skip consistent-operations for newly downloaded realms to avoid double-applying mutations (e.g. @@propmod)

Local intermediate storage

Raw NDJSON buffered locally before atomic apply:

• OPFS in modern browsers (fast, low overhead)
• Dexie $realmDownload table as fallback (CF Workers, older browsers)

Progress reporting

Wire up the existing (currently no-op) progress prop in sync state to emit download progress during realm streaming.

Backward compatible

Opt-in via a flag in the sync request. Old clients/servers fall back to current behavior.

Existing foundations

• StreamingSyncProcessor.ts in dexie-cloud-common — designed but unconnected streaming types and async generators
• registerYDownloadEndpoint.ts — Y.js download already happens outside main transaction
• Export/Import NDJSON streaming — battle-tested streaming infrastructure

Affected repos

• dexie/Dexie.js (dexie-cloud-addon: sync loop, apply, progress)
• dexie/dexie-cloud (server: new streaming endpoint for realm download)
• dexie/dexie-cloud (dexie-cloud-common: protocol types)

References

• Detailed design doc: internal docs/paginated-sync-design.md
• Reported by Bennie Forss (ToToDo) — real production impact with 20k+ objects in CF Durable Objects
• Design discussion: Discord #liz-lead-dev, 2026-03-28

[liz709]

Architecture notes

From a roadmap/architecture perspective, three areas where this reinforces planned features:

• OPFS roadmap (Dexie 5.0): Using OPFS as an intermediate streaming buffer normalizes the OPFS dependency and is a good stepping stone.
• TanStack DB integration: Streaming + progress-events is exactly what reactive UI frameworks need to show meaningful loading state.

One to watch: if we later implement delta sync pagination (natural next step), we want the same 3-step pattern to be reusable. The design appears to be built with that in mind.

On the consistency model

The three-step flow is elegant. One thing worth documenting explicitly:

"Skip consistent-operations for recently-downloaded realms" — technically correct, but creates a small window (between rev1 and rev2) where a new realm could have mutations applied twice under high server load with a large rev-gap.

The system is eventually consistent during initial sync, but converges correctly. This should be documented explicitly for enterprise/GDPR customers — not as a caveat, but as a feature. "Streaming initial sync with guaranteed convergence" is a differentiator vs. competitors.

Recommended ship order

• Ship server-side ASAP — fixes the CF Durable Objects problem now
• Client-side (progress-reporting, OPFS buffering) fits naturally in Dexie 5.x
• Flag-based opt-in keeps backward compatibility: old client + new server = fallback to current behavior

Priority: Critical/High — real production problem from a real customer, addresses the sync-performance gap vs competitors.