feat(index): support xabi scalar index plugins

[Xuanwo]

Summary

This PR adds scalar index plugin support on top of xabi, allowing Lance to load session-scoped index implementations from dynamic libraries without making plugins depend on Lance internals.

xabi is used as the stable ABI layer between Lance and plugins. In this model, #[xabi::data] defines versioned wire data types, and #[xabi::xabi] defines trait contracts that are lowered into C-compatible vtables, handles, borrowed references, futures, and owned byte buffers. Lance and a plugin communicate through those generated ABI contracts instead of Rust trait objects from the same crate build. Larger payloads stay format-oriented: Arrow data crosses the boundary as Arrow IPC, and query expressions cross as Substrait bytes plus host-side JSON context.

The Lance session now owns plugin registrations and exposes scalar index integration through the existing index registry. A fixture mock btree plugin lives outside the workspace as a test-only xabi dynamic library, so the end-to-end dynamic loading path is covered without adding another published crate.

Reviewing the ABI

The concrete ABI should be reviewed in these files:

• rust/lance-index-plugin-abi/src/lib.rs

  • Contract IDs and ABI_VERSION
  • #[xabi::data] request/response structs
  • #[xabi::xabi] traits for plugin, index, training data, index store, and build progress
  • Arrow IPC and little-endian encoding helpers used at the boundary

• rust/lance-index-plugin-abi/xabi/snapshots/lance.*/*.txt

  • Generated xabi layout snapshots for each public contract
  • Concrete field offsets, sizes, alignments, vtable sizes, and contract IDs for supported targets

The host-side adapter is in rust/lance/src/session/xabi_index_plugin.rs, and the test plugin implementation is in rust/lance/tests/xabi_plugins/mock_btree/src/lib.rs.

ABI stability

The ABI stability story is intentionally centered on xabi rather than Lance internals:

• Public contracts use stable lance.* contract IDs and an explicit ABI_VERSION.
• xabi data layouts include size and ABI version fields, allowing prefix-compatible extension.
• xabi vtables track minimum and full sizes so older/newer sides can validate available methods.
• Boundary validation checks ABI version, vtable size, null pointers, and field availability before use.
• xabi-assert snapshots lock the generated ABI layout; intentional ABI changes require updating the snapshots.
• The ABI crate avoids depending on Lance internals, so plugins bind to stable wire contracts instead of private Rust types.

[codecov]

Codecov Report

❌ Patch coverage is 64.67991% with 320 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/session/xabi_index_plugin.rs	57.06%	226 Missing and 26 partials ⚠️
rust/lance-index-plugin-abi/src/lib.rs	44.57%	40 Missing and 6 partials ⚠️
rust/lance/src/index.rs	53.84%	10 Missing and 2 partials ⚠️
rust/lance-index/src/registry.rs	88.52%	7 Missing ⚠️
rust/lance/src/index/scalar.rs	98.43%	0 Missing and 2 partials ⚠️
rust/lance/src/session.rs	92.85%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!