Performance improvements

[jgarzik]

No description provided.

[Copilot]

Pull request overview

This PR introduces a broad set of performance-focused improvements across RooDB’s storage engine and query execution pipeline, and adds end-to-end prepared statement support (COM_STMT_PREPARE / COM_STMT_EXECUTE) to better match MySQL client behavior and sysbench patterns.

Changes:

• Add prepared statement support with plan caching, binary result-set encoding, and placeholder-aware resolve/typecheck/planning.
• Improve LSM read performance via block cache + bloom filters, batched block reads, reader caching, and reduced lock contention.
• Add PK-based fast paths (PointGet/RangeScan) plus INSERT auto_increment handling and related planner/executor updates.

Reviewed changes

Copilot reviewed 54 out of 56 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
tests/roodb_suite/prepared/mod.rs	Adds prepared-statement test module wiring.
tests/roodb_suite/prepared/basic.rs	New integration tests for prepared SELECT/INSERT/UPDATE/DELETE and sysbench-like patterns.
tests/roodb_suite/prepared/auto_increment.rs	Adds AUTO_INCREMENT integration tests (including prepared INSERT).
tests/roodb_suite/mod.rs	Exposes the new prepared test suite.
tests/roodb_suite/cluster/replication.rs	Updates RaftNode constructor calls with single-node flag parameter.
tests/raft_cluster.rs	Updates RaftNode constructor calls with single-node flag parameter.
tests/planner_tests.rs	Asserts planner extracts PK key_value for UPDATE/DELETE point-get fast paths.
tests/executor_tests.rs	Updates tests for PK-key encoding and new plan fields (pk indices / key_value).
src/txn/read_view.rs	Adds all_committed fast-path to MVCC visibility checks.
src/txn/mvcc_storage.rs	Avoids copies in visibility traversal; adds get_with_version() for OCC.
src/txn/manager.rs	Removes per-commit storage flush; optimizes read-view creation allocation behavior.
src/storage/lsm/sstable.rs	SSTable v2: multi-page index, bloom filter in footer, optional block cache, batched scans, eager load.
src/storage/lsm/mod.rs	Exposes new block_cache and bloom modules.
src/storage/lsm/memtable.rs	Raises memtable flush threshold to 64MB.
src/storage/lsm/engine.rs	Adds block cache + reader cache; reduces locking during scans; introduces batched SSTable scan path.
src/storage/lsm/bloom.rs	Implements Bloom filter + tests.
src/storage/lsm/block_cache.rs	Adds LRU block cache implementation + tests.
src/storage/lsm/block.rs	Adds into_entries() to support zero-copy-ish scan paths.
src/sql/typecheck.rs	Treats placeholders as type-compatible to enable prepared-plan templates.
src/sql/resolver.rs	Adds placeholder mode and AUTO_INCREMENT parsing + insert nullability handling.
src/server/listener.rs	Removes TransactionManager storage-flush dependency; tweaks test server leader wait.
src/server/handler.rs	Disables Nagle’s algorithm (TCP_NODELAY) for lower latency.
src/raft/node.rs	Adds single-node Raft config mode; checks state-machine response for application errors; adds perf logs.
src/raft/lsm_storage.rs	Adds single-node mode flush skipping for votes; adds perf logs; refactors OCC apply into two-pass.
src/protocol/roodb/prepared.rs	Implements prepared statement manager, placeholder counting, parameter decoding, and substitution helpers.
src/protocol/roodb/mod.rs	Integrates prepared statements into command loop; adds binary resultsets; adds plan cache usage; buffers IO; adds perf timing.
src/protocol/roodb/metrics.rs	Adds lightweight per-query timing logging utilities.
src/protocol/roodb/handshake.rs	Advertises CLIENT_DEPRECATE_EOF capability.
src/protocol/roodb/command.rs	Parses COM_STMT_EXECUTE raw payload; adds COM_STMT_RESET support.
src/protocol/roodb/binary.rs	Adds binary row encoding for prepared-statement result sets.
src/planner/physical.rs	Adds PointGet/RangeScan plans, prepared param substitution, PK bound extraction, and insert metadata (pk/auto-inc indices).
src/planner/logical/mod.rs	Adds Literal::Placeholder.
src/planner/logical/builder.rs	Propagates placeholder handling into type inference/naming.
src/planner/explain.rs	Adds explain output for PointGet/RangeScan and updates Delete formatting.
src/planner/cost.rs	Adds cost estimates for PointGet/RangeScan.
src/main.rs	Switches to scheduled IO factory; updates RaftNode init signature.
src/executor/update.rs	Adds UPDATE PointGet fast path using OCC version reads.
src/executor/scan.rs	Makes TableScan streaming by lazily decoding rows in next().
src/executor/range_scan.rs	Adds executor to scan PK ranges with tight storage bounds + buffer merge.
src/executor/point_get.rs	Adds executor for single-row PK lookup.
src/executor/mod.rs	Exposes new executors.
src/executor/insert.rs	Adds auto_increment handling, last_insert_id reporting, and PK-based key encoding.
src/executor/engine.rs	Wires new plans/executors and passes insert/update/delete metadata.
src/executor/encoding.rs	Adds PK-based key encoding and comparable datum encoding.
src/executor/delete.rs	Adds DELETE PointGet fast path using OCC version reads.
src/executor/ddl.rs	Ensures DROP TABLE also deletes user data rows for the table.
src/executor/datum.rs	Panics on unsubstituted placeholders to prevent executing unbound plan templates.
src/executor/context.rs	Adds buffered range retrieval to support range scans / filtered buffer merges.
src/catalog/mod.rs	Adds schema_version for prepared plan cache invalidation.
bench/run.sh	Adds sysbench benchmark harness to compare RooDB vs MySQL and emit JSON results.
bench/results/.gitkeep	Keeps benchmark results directory in git.
bench/compare.sh	Adds JSON benchmark comparison utility.
bench/.gitignore	Ignores generated benchmark JSON output.
Cargo.toml	Adds lru dependency for block cache.
Cargo.lock	Locks lru dependency versions (adds lru 0.12.5 alongside existing lru 0.14.0).
CLAUDE.md	Documents benchmarking expectations and policies.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.