Context
The documents data source connector (crates/skardi/src/sources/providers/documents/) currently reads a local directory and re-parses the entire directory into memory on every SELECT (parsing is lazy at query time, with no caching, no streaming, and no pushdown). See the TODO(scale) / TODO(pushdown) comments in table.rs.
Two limitations to track here.
1. Add S3 / object-store input support
- Today the read path is local-filesystem only:
collect_files() uses std::fs::read_dir and parse_file() passes a local path string to LiteParse::parse().
- The
mod.rs doc comment already advertises "directory / object-store prefix" as the intent, and image_store (output side) accepts s3:// URIs but write_image_crop() is a no-op for remote stores — so the aspiration exists but nothing is implemented.
- Goal: let a source
path be an object-store prefix (e.g. s3://bucket/prefix/) so cloud customers can point at their own bucket instead of having to land files on a mounted volume (PVC/hostPath) inside our deployment.
- Scope to consider: listing objects under a prefix, fetching bytes for parse, and finishing the deferred
image_store upload path for remote stores.
2. Handle the full-directory scan case (predicate / prefix pushdown)
TableProvider::scan() currently ignores its _filters: &[Expr] argument and does not implement supports_filters_pushdown, so DataFusion applies any WHERE above the scan as a separate FilterExec. A query like WHERE path LIKE 'batch-a/%' still parses the whole directory; even LIMIT 1 parses everything and then truncates.
- Goal: implement
supports_filters_pushdown for path-prefix predicates and push the prefix into collect_files() (prune the walk, ideally only read_dir the matching subtree). Also consider limit pushdown and per-file streaming (RecordBatchStream yielding as each file parses) so large sources don't block a tokio worker while materializing one big batch.
Notes / references
crates/skardi/src/sources/providers/documents/table.rs — scan() (ignored filters), fetch_partition() (eager parse, TODO(scale)/TODO(pushdown))
crates/skardi/src/sources/providers/documents/parse.rs — collect_files(), parse_file(), write_image_crop() (remote no-op)
crates/skardi/src/sources/providers/documents/mod.rs — "directory / object-store prefix" doc comment
Context
The
documentsdata source connector (crates/skardi/src/sources/providers/documents/) currently reads a local directory and re-parses the entire directory into memory on everySELECT(parsing is lazy at query time, with no caching, no streaming, and no pushdown). See theTODO(scale)/TODO(pushdown)comments intable.rs.Two limitations to track here.
1. Add S3 / object-store input support
collect_files()usesstd::fs::read_dirandparse_file()passes a local path string toLiteParse::parse().mod.rsdoc comment already advertises "directory / object-store prefix" as the intent, andimage_store(output side) acceptss3://URIs butwrite_image_crop()is a no-op for remote stores — so the aspiration exists but nothing is implemented.pathbe an object-store prefix (e.g.s3://bucket/prefix/) so cloud customers can point at their own bucket instead of having to land files on a mounted volume (PVC/hostPath) inside our deployment.image_storeupload path for remote stores.2. Handle the full-directory scan case (predicate / prefix pushdown)
TableProvider::scan()currently ignores its_filters: &[Expr]argument and does not implementsupports_filters_pushdown, so DataFusion applies anyWHEREabove the scan as a separateFilterExec. A query likeWHERE path LIKE 'batch-a/%'still parses the whole directory; evenLIMIT 1parses everything and then truncates.supports_filters_pushdownforpath-prefix predicates and push the prefix intocollect_files()(prune the walk, ideally onlyread_dirthe matching subtree). Also considerlimitpushdown and per-file streaming (RecordBatchStreamyielding as each file parses) so large sources don't block a tokio worker while materializing one big batch.Notes / references
crates/skardi/src/sources/providers/documents/table.rs—scan()(ignored filters),fetch_partition()(eager parse,TODO(scale)/TODO(pushdown))crates/skardi/src/sources/providers/documents/parse.rs—collect_files(),parse_file(),write_image_crop()(remote no-op)crates/skardi/src/sources/providers/documents/mod.rs— "directory / object-store prefix" doc comment