[Feature] Native support for ColBERT-style late interaction retrieval

[reallyticsai]

Description

ColBERT-style late interaction models offer a great tradeoff between single-vector and cross-encoder approaches. Native support in txtai would give users another powerful retrieval option.

Benefits

• Better recall than single-vector for complex queries
• Much faster than cross-encoder at retrieval time
• Proven effectiveness in recent benchmarks (BEIR, MTEB)

[davidmezzetti]

Hello. Currently txtai supports multi-vector search through MUVERA.

There is also this issue that will use the LEMUR algorithm.

If you want to search multi-vectors directly with the current setup, you can use the reranker pipeline.

[cschanhniem]

ColBERT-style late interaction would be a significant architectural addition to txtai, not just a new embedding backend. A few technical considerations based on working with multi-vector retrieval:

Scoring pathway vs. embedding swap. ColBERT\s MaxSim operator computes \sum_{i} \max_{j} (q_i · d_j) — a two-step similarity that needs per-token embeddings accessible at query time. This is fundamentally different from txtai\s current pipeline, which computes a single vector per document. Implementing this means adding a deferred similarity path: store per-token/document vectors (Compressed Residual Representations), compute MaxSim at query time, then merge with existing vector/BM25 results for hybrid search. PluggableScoringStrategy in the existing workflow is the right extension point.

Storage tradeoff. A ColBERT model with 128 dims per token at sequence length 512 produces ~65K floats per document. Using FP16, that is ~130KB per document vs. ~2KB for single-vector embeddings. For a 1M document corpus, that is ~130GB of token-level embeddings. IVF-PQ compression (as used in the ColBERTv2 implementation) brings this down to ~10GB but requires integrating a residual compression step in the indexing pipeline.

Connection to LEMUR (#1024). Both ColBERT and LEMUR are multi-vector approaches, but they optimize different parts of the MaxSim bottleneck. ColBERT compresses per-document embeddings (residual compression), while LEMUR learns a fixed set of cluster centroids to avoid storing per-document embeddings entirely. LEMUR is better for write-optimized workloads; ColBERT is better for freshness-sensitive retrieval where index rebuild overhead matters. Ideally txtai would support both under a common late-interaction abstraction.

txtai\s advantage for this feature. Because txtai already supports configurable backends and composable pipelines, the late-interaction path could be exposed as just another VectorDatabase implementation with a custom similarity() override for MaxSim. The hard part is the storage layer — existing backends (Faiss, Annoy, HNSWLib) do not natively support multi-vector per-document. Would a dedicated late-interaction storage layer be the right approach, or is the plan to extend an existing backend?