Follow-up from PR #158 review (#158 (comment)).
Two avoidable copies on the ad-hoc query result path:
batches_to_single (crates/skardi/src/engine/datafusion.rs) uses concat_batches to deep-copy every column into one contiguous batch, which the handler then slices to max_rows.record_batch_to_json (crates/server/src/response.rs) encodes arrow → JSON bytes → re-parses into a serde_json::Value tree → the envelope re-serializes it — three passes over client-controlled-size data.
Proposed fix: return Vec<RecordBatch> from execute_with_limit, apply the row cap across batches without concatenation, and feed the batches straight to arrow's JSON writer (it accepts multiple batches), embedding the resulting bytes via serde_json::value::RawValue. Requires touching the shared response envelope used by pipeline responses, hence a separate change.
🤖 Generated with Claude Code
Follow-up from PR #158 review (#158 (comment)).
Two avoidable copies on the ad-hoc query result path:
batches_to_single(crates/skardi/src/engine/datafusion.rs) usesconcat_batchesto deep-copy every column into one contiguous batch, which the handler then slices tomax_rows.record_batch_to_json(crates/server/src/response.rs) encodes arrow → JSON bytes → re-parses into aserde_json::Valuetree → the envelope re-serializes it — three passes over client-controlled-size data.Proposed fix: return
Vec<RecordBatch>fromexecute_with_limit, apply the row cap across batches without concatenation, and feed the batches straight to arrow's JSON writer (it accepts multiple batches), embedding the resulting bytes viaserde_json::value::RawValue. Requires touching the shared response envelope used by pipeline responses, hence a separate change.🤖 Generated with Claude Code