docs(cayenne): datalake promotion is incremental carry-forward, not whole-table#1920
Open
claudespice wants to merge 1 commit into
Open
docs(cayenne): datalake promotion is incremental carry-forward, not whole-table#1920claudespice wants to merge 1 commit into
claudespice wants to merge 1 commit into
Conversation
…hole-table spiceai/spiceai#11745 rewrote cold-tier (datalake) promotion from whole-table re-materialization to incremental carry-forward: - dirty/clean manifest classification via per-PK-column stat rectangles refined by per-file PK bloom filters (conservative in the safe direction) - clean files carried forward by manifest reference, never re-read - per-file PK blooms serve upsert cold-tier key existence without a scan - periodic mark-and-sweep physical GC (cayenne_datalake_gc_interval_ms, default 5m, doubling as the orphan grace period) The published How-it-works section still described whole-table re-materialization; correct the promotion + read-path bullets and add a physical-GC bullet to match current trunk.
Contributor
✅ Pull with Spice PassedPassing checks:
|
Contributor
🔍 Pull with Spice FailedPassing checks:
Failed checks:
Please address these issues and update your pull request. |
Contributor
|
🚀 deployed to https://9b3b7b95.spiceai-org-website.pages.dev |
lukekim
approved these changes
Jul 13, 2026
lukekim
approved these changes
Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Cayenne Cold Object-Store Tier "How it works" section described promotion as re-materializing the whole visible table each cycle. spiceai/spiceai#11745 (merged, on
trunkas6e0751ae8) rewrote datalake (cold) tier promotion to incremental carry-forward, so the published description is now stale.Corrected against current
trunksource (crates/cayenne/src/provider/cold_partition.rs,pk_index.rs):cayenne_datalake_gc_interval_ms(default 5m), which doubles as the orphan grace period.Datalake/cold-tier is vNext-only (params exist only under
website/docs/, notversioned_docs/), so no versioned propagation is needed. Prose-only diff — no new links or tags.Source PRs
Test plan
cd website && npm run buildpasses (Docusaurus;Generated static files, exit 0)website/docs/components/data-accelerators/cayenne/index.md)