Skip to content

documents connector: add object-store (S3) input + predicate/prefix pushdown for scans #167

Description

@abbccdda

Context

The documents data source connector (crates/skardi/src/sources/providers/documents/) currently reads a local directory and re-parses the entire directory into memory on every SELECT (parsing is lazy at query time, with no caching, no streaming, and no pushdown). See the TODO(scale) / TODO(pushdown) comments in table.rs.

Two limitations to track here.

1. Add S3 / object-store input support

  • Today the read path is local-filesystem only: collect_files() uses std::fs::read_dir and parse_file() passes a local path string to LiteParse::parse().
  • The mod.rs doc comment already advertises "directory / object-store prefix" as the intent, and image_store (output side) accepts s3:// URIs but write_image_crop() is a no-op for remote stores — so the aspiration exists but nothing is implemented.
  • Goal: let a source path be an object-store prefix (e.g. s3://bucket/prefix/) so cloud customers can point at their own bucket instead of having to land files on a mounted volume (PVC/hostPath) inside our deployment.
  • Scope to consider: listing objects under a prefix, fetching bytes for parse, and finishing the deferred image_store upload path for remote stores.

2. Handle the full-directory scan case (predicate / prefix pushdown)

  • TableProvider::scan() currently ignores its _filters: &[Expr] argument and does not implement supports_filters_pushdown, so DataFusion applies any WHERE above the scan as a separate FilterExec. A query like WHERE path LIKE 'batch-a/%' still parses the whole directory; even LIMIT 1 parses everything and then truncates.
  • Goal: implement supports_filters_pushdown for path-prefix predicates and push the prefix into collect_files() (prune the walk, ideally only read_dir the matching subtree). Also consider limit pushdown and per-file streaming (RecordBatchStream yielding as each file parses) so large sources don't block a tokio worker while materializing one big batch.

Notes / references

  • crates/skardi/src/sources/providers/documents/table.rsscan() (ignored filters), fetch_partition() (eager parse, TODO(scale)/TODO(pushdown))
  • crates/skardi/src/sources/providers/documents/parse.rscollect_files(), parse_file(), write_image_crop() (remote no-op)
  • crates/skardi/src/sources/providers/documents/mod.rs — "directory / object-store prefix" doc comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions