Tracks two follow-ups flagged after PR #122 landed the SQL INSERT path through ChromaInsertExec.
1. No bulk vectors-only entry point
Today the only way to insert into a Chroma collection is via SQL INSERT INTO collection VALUES (id, document, embedding, metadata), which routes through DataFusion's planner and ChromaInsertExec (crates/skardi/src/sources/providers/chroma/writes.rs:121-188).
For workloads that just want to push raw (id, vector) pairs in bulk (e.g. ETL from a parquet file of embeddings), this requires constructing a row plan with all four columns even when document and metadata are absent, and pays the per-row Arrow conversion cost in extract_insert_columns. Worth considering a thinner ingestion API — e.g. a chroma_bulk_insert(table, ids[], vectors[][]) UDTF or an HTTP-side bulk endpoint — that bypasses the SQL planner and streams directly to ChromaCollection::add in larger chunks.
2. No live-server integration tests
Only the conversion / filter logic has unit tests (filter.rs — 12 tests). The end-to-end paths through ChromaScanExec, ChromaKnnExec (literal + deferred-vector), ChromaInsertExec (Append + Replace), and ChromaDmlExec::Delete have never been exercised against an actual Chroma server. Lance has the same #[ignore]-gated integration test pattern (lance/knn_table_function.rs:288) — we should mirror it: spin up Chroma in docker-compose, seed a small collection, and gate the tests behind --ignored so CI doesn't require the service to be up.
Files / lines
crates/skardi/src/sources/providers/chroma/writes.rs:121-188 — INSERT execcrates/skardi/src/sources/providers/chroma/writes.rs:295-366 — column extractioncrates/skardi/src/sources/providers/chroma/knn_exec.rs — literal + deferred KNNcrates/skardi/src/sources/providers/chroma/scan_exec.rs — scan path
Tracks two follow-ups flagged after PR #122 landed the SQL
INSERTpath throughChromaInsertExec.1. No bulk vectors-only entry point
Today the only way to insert into a Chroma collection is via SQL
INSERT INTO collection VALUES (id, document, embedding, metadata), which routes through DataFusion's planner andChromaInsertExec(crates/skardi/src/sources/providers/chroma/writes.rs:121-188).For workloads that just want to push raw
(id, vector)pairs in bulk (e.g. ETL from a parquet file of embeddings), this requires constructing a row plan with all four columns even whendocumentandmetadataare absent, and pays the per-row Arrow conversion cost inextract_insert_columns. Worth considering a thinner ingestion API — e.g. achroma_bulk_insert(table, ids[], vectors[][])UDTF or an HTTP-side bulk endpoint — that bypasses the SQL planner and streams directly toChromaCollection::addin larger chunks.2. No live-server integration tests
Only the conversion / filter logic has unit tests (
filter.rs— 12 tests). The end-to-end paths throughChromaScanExec,ChromaKnnExec(literal + deferred-vector),ChromaInsertExec(Append + Replace), andChromaDmlExec::Deletehave never been exercised against an actual Chroma server. Lance has the same#[ignore]-gated integration test pattern (lance/knn_table_function.rs:288) — we should mirror it: spin up Chroma in docker-compose, seed a small collection, and gate the tests behind--ignoredso CI doesn't require the service to be up.Files / lines
crates/skardi/src/sources/providers/chroma/writes.rs:121-188— INSERT execcrates/skardi/src/sources/providers/chroma/writes.rs:295-366— column extractioncrates/skardi/src/sources/providers/chroma/knn_exec.rs— literal + deferred KNNcrates/skardi/src/sources/providers/chroma/scan_exec.rs— scan path