bug: compaction fails determinstically on a very small-fragment dataset

[mackrorysd]

Description

Invalid user input: projection supplied fewer column indices than its fields require (ran out at field '..'), at rust/lance-file/src/versions/v2_0/reader.rs:249:13. The dataset is a case of 1 row-per-fragment, and everycompaction fails within ~1.5 seconds, so the fragmentation continues. The error originates in validate_field_length, which walks the projected schema and consumes one entry from the projection's column_indices per value-bearing field; for at least one data file in the dataset, the projection supplies fewer column indices than the schema walk requires, running out at the field named in the error. Working hypothesis is a schema-evolution mismatch: some fragments' data files were written under an earlier schema (before a later-added column), and the ReaderProjection built from the current dataset schema doesn't line up with those files' physical column layout. Observing this on both lance 11.0.0-beta.22 and 12.0.0-beta.16.

Steps to reproduce

Can work to provide the offending data set

Expected behavior

No response

Lance version

11.0.0-beta.22

Language binding

Rust

Environment

No response

Logs / traceback