Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions website/docs/components/data-accelerators/cayenne/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -612,8 +612,9 @@ The cold tier lets a table grow beyond local NVMe capacity while keeping recent,

### How it works

- **Promotion (write path).** A dedicated background worker graduates the warm tier once the size or file-count threshold (`cayenne_cold_tier_warm_max_bytes` / `cayenne_cold_tier_warm_max_files`) is crossed, evaluated every `cayenne_cold_tier_background_interval_ms`. Promotion re-materializes the visible table (all deletes applied, one version per key), **Z-order clusters** it for tight multi-column zone maps, writes read-optimized Vortex at the larger `cayenne_cold_target_file_size_mb`, and atomically registers the cold files while clearing the promoted warm files in a single transaction.
- **Cross-tier scan (read path).** Queries span all tiers and push filters, projection, and limits down to each. Cold files are pruned from statistics held in the metastore, so pruning requires **no object-store round-trip on the query path**. A `DELETE` after promotion correctly hides a cold-resident row.
- **Promotion (write path).** A dedicated background worker graduates the warm tier once the size or file-count threshold (`cayenne_cold_tier_warm_max_bytes` / `cayenne_cold_tier_warm_max_files`) is crossed, evaluated every `cayenne_cold_tier_background_interval_ms`. Promotion is **incremental carry-forward**: rather than re-materializing the whole table each cycle, it classifies the existing cold manifest into *dirty* files that may host a tombstoned key — using per-PK-column min/max rectangles from each file's persisted statistics, refined by a per-file PK bloom filter, and conservative in the safe direction (a false positive costs an extra rewrite; a missed tombstone is impossible) — and *clean* files that are provably untouched. Only the warm delta plus the dirty files are re-read (all deletes applied, one version per key), **Z-order clustered** for tight multi-column zone maps, and written as read-optimized Vortex at the larger `cayenne_cold_target_file_size_mb` under a per-promotion prefix; clean files are carried forward by manifest reference and never re-read. The new files are atomically registered while the promoted warm files are cleared in a single transaction, so promotion cost tracks the *changed* data, not total table size.
- **Cross-tier scan (read path).** Queries span all tiers and push filters, projection, and limits down to each. Cold files are pruned from statistics held in the metastore, so pruning requires **no object-store round-trip on the query path**. Each cold file's PK bloom lets an upsert's keyset rebuild answer cold-tier key existence without scanning the object store. A `DELETE` after promotion correctly hides a cold-resident row.
- **Physical GC.** A periodic mark-and-sweep, rooted at the manifest, reclaims cold objects orphaned by carry-forward rewrites. It runs every `cayenne_datalake_gc_interval_ms` (default 5m), which doubles as the orphan grace period: an object no longer referenced by the manifest is deleted only after it has been observed orphaned for at least one interval.
- **Clustering.** Cold files are clustered by `cayenne_datalake_clustering_columns` (multi-column Z-order / Morton order), falling back to `cayenne_sort_columns` and then the primary key. Clustering on more than one dimension prunes far better than a single-column sort for selective queries on any clustering column.

### Requirements and v1 constraints
Expand Down
Loading