A Delete that rebases over a concurrently committed DataOverlay silently discards the overlay. The overlaid cell reverts to its base value, which a reader cannot distinguish from the overlay write never having happened.
Reproduction
On main (verified at 8cdffd30e), through the public write paths only — DeleteBuilder::execute_uncommitted, FileFragment::write_overlay, CommitBuilder:
- Create a dataset; call its current version
v.
- Stage a delete against
v with DeleteBuilder::execute_uncommitted, targeting a row that is not the overlaid one.
- Commit a
DataOverlay on the same fragment, supplying a new value for a surviving row. It lands at v+1.
- Commit the staged delete against
v. It rebases over the overlay and lands at v+2.
At v+1 a scan returns the overlaid value. At v+2 it returns the base value, and the fragment has zero overlays.
Cause
Operation::Delete's apply replaces the fragment entry wholesale from a post-image built at the read version. Unlike Operation::Update's arm, which does updated.overlays = f.overlays.clone(), the Delete arm does not carry overlays forward. Meanwhile check_delete_txn explicitly permits a concurrent DataOverlay, so the pair is allowed to land.
Ordering-dependent: overlay-after-delete is fine; it is only delete-rebased-over-overlay that loses data.
Coverage
A failing test asserting the correct behaviour is committed #[ignore]d as io::commit::conflict_matrix::cases::delete_must_not_drop_a_concurrent_overlay, pointing at this issue. The (Delete, DataOverlay) cell is excluded from that module's matrix until this is fixed.
cargo test -p lance --lib conflict_matrix -- --ignored
A
Deletethat rebases over a concurrently committedDataOverlaysilently discards the overlay. The overlaid cell reverts to its base value, which a reader cannot distinguish from the overlay write never having happened.Reproduction
On
main(verified at8cdffd30e), through the public write paths only —DeleteBuilder::execute_uncommitted,FileFragment::write_overlay,CommitBuilder:v.vwithDeleteBuilder::execute_uncommitted, targeting a row that is not the overlaid one.DataOverlayon the same fragment, supplying a new value for a surviving row. It lands atv+1.v. It rebases over the overlay and lands atv+2.At
v+1a scan returns the overlaid value. Atv+2it returns the base value, and the fragment has zero overlays.Cause
Operation::Delete's apply replaces the fragment entry wholesale from a post-image built at the read version. UnlikeOperation::Update's arm, which doesupdated.overlays = f.overlays.clone(), theDeletearm does not carryoverlaysforward. Meanwhilecheck_delete_txnexplicitly permits a concurrentDataOverlay, so the pair is allowed to land.Ordering-dependent: overlay-after-delete is fine; it is only delete-rebased-over-overlay that loses data.
Coverage
A failing test asserting the correct behaviour is committed
#[ignore]d asio::commit::conflict_matrix::cases::delete_must_not_drop_a_concurrent_overlay, pointing at this issue. The(Delete, DataOverlay)cell is excluded from that module's matrix until this is fixed.