Follow-up from PR #142.
Background
The InfluxDB 3 provider registers a FlightTable whose SQL is fixed at registration time (SELECT * FROM "<measurement>"). The vendored datafusion-table-providers Flight provider's scan ignores the filters/limit DataFusion hands it (flight.rs:333), and there is no flight-federation feature (only postgres/mysql/sqlite/etc. have *-federation).
Consequence: any predicate/projection/limit that comes from the pipeline SQL is applied in Skardi after the full measurement is transferred over Flight. InfluxDB only ever sees a full scan. This is fine for small/bounded measurements but is a full-scan-per-query cost for large time-series.
The static query option is the only current pushdown lever (its SQL is sent verbatim and runs inside InfluxDB), but it's fixed per data source, not derived per request.
Proposal (Option 1 — SQL rewrite in the wrapper)
Make CountSafeFlightTable::scan the pushdown point. Store the factory + endpoint + base query; at scan time, translate the requested projection/filters/limit into SQL (DataFusion Unparser for Expr -> SQL, wrapping the base query as a subquery), then open a fresh Flight table with that query. Since InfluxDB 3 is DataFusion, unparse fidelity is high.
Downsides to design around (from PR #142 discussion)
- Correctness under engine-version divergence (serious). Skardi's DataFusion (52.1) vs InfluxDB's bundled version may differ in coercion/null/timestamp/timezone/UDF semantics.
Inexact pushdown protects against over-fetching, not under-fetching — a stricter-than-intended remote predicate silently drops valid rows and local re-filtering can't recover them. → Push only a conservative whitelist of expr shapes (col <op> literal, IN, IS NULL, boolean combinations); never casts/datetime-arithmetic/UDFs.
- Robustness. A rejected rewritten query turns a previously-working pipeline into an error. → Fall back to the base query + local filter on any rewrite/open failure.
- Schema-shape matching. The rewritten plan's Arrow schema (types + nullability) must exactly match the requested projection, or it errors/panics (same class as the
count(*)/enforce_schema bug).
- Per-scan RPC overhead. Re-opening re-runs
GetFlightInfo; for many small queries the extra round-trip can offset transfer savings. → consider caching the metadata supplier.
- Subquery wrapping fragility with custom
query base queries (aliases, GROUP BY, computed columns). → gate filter pushdown on a plain-scan base query.
Suggested staging
- Phase 1: projection + limit pushdown (low risk, real wins).
- Phase 2: filter pushdown, whitelisted +
Inexact + fallback.
- (Later / alternative) Option 2: a proper
datafusion-federation SQLExecutor over Flight SQL, which would also push aggregations/joins into InfluxDB (note: crate's federation integration targets datafusion-federation 0.4 while Skardi pins =0.5.1).
Follow-up from PR #142.
Background
The InfluxDB 3 provider registers a
FlightTablewhose SQL is fixed at registration time (SELECT * FROM "<measurement>"). The vendoreddatafusion-table-providersFlight provider'sscanignores thefilters/limitDataFusion hands it (flight.rs:333), and there is noflight-federationfeature (only postgres/mysql/sqlite/etc. have*-federation).Consequence: any predicate/projection/limit that comes from the pipeline SQL is applied in Skardi after the full measurement is transferred over Flight. InfluxDB only ever sees a full scan. This is fine for small/bounded measurements but is a full-scan-per-query cost for large time-series.
The static
queryoption is the only current pushdown lever (its SQL is sent verbatim and runs inside InfluxDB), but it's fixed per data source, not derived per request.Proposal (Option 1 — SQL rewrite in the wrapper)
Make
CountSafeFlightTable::scanthe pushdown point. Store the factory + endpoint + base query; at scan time, translate the requested projection/filters/limit into SQL (DataFusionUnparserforExpr-> SQL, wrapping the base query as a subquery), then open a fresh Flight table with that query. Since InfluxDB 3 is DataFusion, unparse fidelity is high.Downsides to design around (from PR #142 discussion)
Inexactpushdown protects against over-fetching, not under-fetching — a stricter-than-intended remote predicate silently drops valid rows and local re-filtering can't recover them. → Push only a conservative whitelist of expr shapes (col <op> literal,IN,IS NULL, boolean combinations); never casts/datetime-arithmetic/UDFs.count(*)/enforce_schemabug).GetFlightInfo; for many small queries the extra round-trip can offset transfer savings. → consider caching the metadata supplier.querybase queries (aliases, GROUP BY, computed columns). → gate filter pushdown on a plain-scan base query.Suggested staging
Inexact+ fallback.datafusion-federationSQLExecutorover Flight SQL, which would also push aggregations/joins into InfluxDB (note: crate's federation integration targets datafusion-federation 0.4 while Skardi pins =0.5.1).