Skip to content

Write Range and Delta offsets in Lance 2.3 #8008

Description

@Xuanwo

Goal

Use Range and Delta block compression for Lance 2.3 variable-width mini-block offsets when it reduces complete chunk size.

Scope

  • Compare legacy interleaved-flat and generic offset containers.
  • Keep the legacy candidate when generic metadata or payload placement is not beneficial.
  • Preserve Lance 2.0-2.2 bytes, SparseLayout wire, and chunk-local random-read behavior.
  • Add representative encode, scan, range, take, and multi-chunk tests and benchmarks.

Context

This is PR 5 of the Generic Block Compression stack extracted from #8002. It depends on the generic offset reader.

Metadata

Metadata

Assignees

Labels

A-encodingEncoding, IO, file reader/writerA-formatOn-disk format: protos and format spec docsenhancementNew feature or requestformat-changeA change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo).

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions