`optimize_indices` cannot extend a scalar index created over a nested field of an empty dataset

[jennifermell]

Description

create_scalar_index("st.child", index_type="BTREE") succeeds whether or not the dataset has rows. But if it ran while the dataset was empty, the optimize_indices() that should later extend the index over newly appended fragments fails:

OSError: Invalid user input: No column with name child,
/rust/lance/src/index/scalar.rs:364:20

The same index built once rows exist optimizes fine, then and on every later pass.

Output of the repro script:

index built while empty:  optimize_indices() -> OSError: Invalid user input: No column with name child, .../rust/lance/src/index/scalar.rs:364:20
index built over 8 rows:  optimize_indices() -> OK

Steps to reproduce

import tempfile
from pathlib import Path

import lance
import pyarrow as pa

STRUCT = pa.struct([pa.field("child", pa.int64())])
SCHEMA = pa.schema([pa.field("uid", pa.string()), pa.field("st", STRUCT)])


def rows(n: int, start: int = 0) -> pa.Table:
    return pa.table(
        {
            "uid": pa.array([f"u{i}" for i in range(start, start + n)]),
            "st": pa.array([{"child": i} for i in range(start, start + n)], type=STRUCT),
        },
        schema=SCHEMA,
    )


def run(label: str, rows_before_index: int) -> None:
    uri = str(Path(tempfile.mkdtemp()) / "ds.lance")
    initial = SCHEMA.empty_table() if rows_before_index == 0 else rows(rows_before_index)
    lance.write_dataset(initial, uri, schema=SCHEMA, mode="create", data_storage_version="2.2")

    lance.dataset(uri).create_scalar_index("st.child", index_type="BTREE")
    lance.write_dataset(rows(4, start=100), uri, mode="append")  # a fragment the index misses

    try:
        lance.dataset(uri).optimize.optimize_indices()
    except Exception as exc:
        print(f"{label}  optimize_indices() -> {type(exc).__name__}: {exc}")
    else:
        print(f"{label}  optimize_indices() -> OK")


run("index built while empty:", rows_before_index=0)
run("index built over 8 rows:", rows_before_index=8)

Expected behavior

Both cases reach full coverage: an index over a nested field should be extendable regardless of whether the dataset held rows when the index was created.

Lance version

11.0.0

Language binding

Python

Environment

macOS 26.6.2, arm64, local

Logs / traceback