Skip to content

docs(cayenne): datalake promotion is incremental carry-forward, not whole-table#1920

Open
claudespice wants to merge 1 commit into
trunkfrom
cayenne-incremental-datalake-promotion
Open

docs(cayenne): datalake promotion is incremental carry-forward, not whole-table#1920
claudespice wants to merge 1 commit into
trunkfrom
cayenne-incremental-datalake-promotion

Conversation

@claudespice

Copy link
Copy Markdown
Collaborator

Summary

The Cayenne Cold Object-Store Tier "How it works" section described promotion as re-materializing the whole visible table each cycle. spiceai/spiceai#11745 (merged, on trunk as 6e0751ae8) rewrote datalake (cold) tier promotion to incremental carry-forward, so the published description is now stale.

Corrected against current trunk source (crates/cayenne/src/provider/cold_partition.rs, pk_index.rs):

  • Promotion (write path) — now incremental carry-forward: the cold manifest is classified into dirty files (may host a tombstoned key — detected via per-PK-column min/max rectangles from persisted stats, refined by a per-file PK bloom, conservative in the safe direction) and clean files carried forward by manifest reference and never re-read. Only the warm delta plus dirty files are re-read, Z-order clustered, and written under a per-promotion prefix — so cost tracks changed data, not table size.
  • Cross-tier scan (read path) — added that each cold file's PK bloom lets an upsert keyset rebuild answer cold-tier key existence without an object-store scan.
  • Physical GC — new bullet: periodic manifest-rooted mark-and-sweep reclaims objects orphaned by carry-forward rewrites, every cayenne_datalake_gc_interval_ms (default 5m), which doubles as the orphan grace period.

Datalake/cold-tier is vNext-only (params exist only under website/docs/, not versioned_docs/), so no versioned propagation is needed. Prose-only diff — no new links or tags.

Source PRs

Test plan

  • cd website && npm run build passes (Docusaurus; Generated static files, exit 0)
  • Versioned-docs propagation checked — vNext-only feature, no versioned copies exist
  • Files updated: 1 (website/docs/components/data-accelerators/cayenne/index.md)

…hole-table

spiceai/spiceai#11745 rewrote cold-tier (datalake) promotion from
whole-table re-materialization to incremental carry-forward:
- dirty/clean manifest classification via per-PK-column stat rectangles
  refined by per-file PK bloom filters (conservative in the safe direction)
- clean files carried forward by manifest reference, never re-read
- per-file PK blooms serve upsert cold-tier key existence without a scan
- periodic mark-and-sweep physical GC (cayenne_datalake_gc_interval_ms,
  default 5m, doubling as the orphan grace period)

The published How-it-works section still described whole-table
re-materialization; correct the promotion + read-path bullets and add a
physical-GC bullet to match current trunk.
@github-actions

github-actions Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

✅ Pull with Spice Passed

Passing checks:

  • ✅ Title meets minimum length requirement (10 characters)
  • ✅ Has at least one of the required labels: area/blog, area/docs, area/cookbook, dependencies
  • ✅ No banned labels detected
  • ✅ Has at least one assignee: claudespice

@github-actions

Copy link
Copy Markdown
Contributor

🔍 Pull with Spice Failed

Passing checks:

  • ✅ Title meets minimum length requirement (10 characters)
  • ✅ Has at least one of the required labels: area/blog, area/docs, area/cookbook, dependencies
  • ✅ No banned labels detected

Failed checks:

  • ❌ At least one assignee is required for this pull request.

Please address these issues and update your pull request.

@github-actions

Copy link
Copy Markdown
Contributor

🚀 deployed to https://9b3b7b95.spiceai-org-website.pages.dev

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants