perf(sqlite): encode nested JSONB values linearly

[kishansaaai]

Summary

Closes #5192.

Optimizes SQLite JSONB serialization (ToSql<sql_types::Jsonb, Sqlite> for serde_json::Value) by replacing the previous quadratic nested-buffer copying with a linear, two-pass iterative write plan. Also introduces a Criterion-based benchmark suite (benches/sqlite_jsonb.rs) to track scaling across depth and size.

Motivation & Background

Previously, encoding nested JSON arrays and objects allocated a new Vec<u8> for every container level. When a nested container completed, its entire serialized buffer was copied into its parent container. For structures of depth $d$, this caused quadratic $O(d^2)$ buffer allocation and copying ($1 + 2 + \dots + d$ bytes).

As discussed in #5192 (following #5172), we can avoid intermediate buffer allocations and repeated copying by computing container sizes iteratively in advance and writing every header and payload directly into the final destination buffer.

Solution

The new encoding routine works in two linear phases without recursion:

1. Iterative Planning (Leaves to Root flattening):
   Traverse the serde_json::Value iteratively using an explicit visits stack to prevent stack overflow on deep inputs. Each scalar or container is recorded into a sequential write_items plan (JsonbWriteItem::Scalar or JsonbWriteItem::Composite). Scalar payloads borrow their byte slices when possible (or store owned/escaped representations) without allocating intermediate container byte buffers.

2. Bottom-Up Size Calculation & Direct Write:
   Iterate backwards over the plan (from leaves up to root) using an encoded_sizes stack to compute exact container payload sizes and header lengths with checked arithmetic.
   Reserve the total calculated capacity in the destination buffer once (buffer.try_reserve(encoded_size)).
   Perform a final forward pass over write_items, writing each header and payload into the reserved buffer exactly once.

Benchmark Results (Criterion)

A Criterion benchmark suite (benches/sqlite_jsonb.rs) was added comparing main and this branch across depth and size scaling:

1. Depth Scaling — Nested Arrays ([[[[... 1 ...]]]])

Depth	main (baseline)	This Branch	Speedup / Improvement
10	2.05 µs	0.83 µs	2.45x faster (-59%)
50	10.62 µs	2.32 µs	4.58x faster (-78%)
100	29.51 µs	4.08 µs	7.23x faster (-86%)
250	77.54 µs	10.58 µs	7.33x faster (-86%)
500	195.70 µs	19.94 µs	9.81x faster (-90%)
1000	409.44 µs	40.11 µs	10.21x faster (-90%)
2000	924.53 µs	75.43 µs	12.26x faster (-92%)

2. Depth Scaling — Nested Objects ({"k": {"k": ...}})

Depth	main (baseline)	This Branch	Speedup / Improvement
10	5.05 µs	1.92 µs	2.63x faster (-62%)
50	25.20 µs	5.96 µs	4.23x faster (-76%)
100	49.32 µs	11.14 µs	4.43x faster (-77%)
250	157.27 µs	25.45 µs	6.18x faster (-84%)
500	343.44 µs	46.56 µs	7.38x faster (-86%)
1000	707.05 µs	99.91 µs	7.08x faster (-86%)

3. Size Scaling — Flat Array of Objects ([{"id": i, "name": ...}])

Elements	main (baseline)	This Branch	Speedup / Improvement
10	9.92 µs	5.36 µs	1.85x faster (-46%)
100	134.16 µs	30.06 µs	4.46x faster (-78%)
1000	1.83 ms	0.30 ms	6.10x faster (-84%)
5000	4.73 ms	2.06 ms	2.30x faster (-56%)

4. Size Scaling — Flat Array of Scalars ([0, 1, 2, ..., N])

Elements	main (baseline)	This Branch	Speedup / Improvement
10	2.43 µs	2.50 µs	Identical
100	19.00 µs	18.14 µs	1.05x faster
1000	171.40 µs	171.25 µs	Identical
10000	1.78 ms	1.72 ms	1.03x faster

Testing Performed

• Unit tests: 58 targeted SQLite JSON and JSONB tests passed (including upstream's negative integer fix test check_signed_integer).
• Criterion suite: Added and verified cargo bench -p diesel --bench sqlite_jsonb --features "sqlite serde_json".
• Stress & Nesting validation: Verified encoding on deeply nested 2,000-level arrays, objects, and mixed structures without stack overflow or performance degradation.
• Style & Lints: Passed cargo clippy -p diesel --bench sqlite_jsonb with warnings denied (-D warnings) and cargo fmt --check.
• Changelog: Included entry in CHANGELOG.md under ### Changed.

AI-Use Disclosure

In accordance with Diesel's CONTRIBUTING.md and NLnet foundation guidelines on generative AI:

• An AI coding assistant (Gemini / Antigravity) was used to assist in brainstorming the two-pass write-plan design, drafting the bottom-up size calculation arithmetic, generating test fixtures, and setting up the Criterion benchmark harness.
• All code, memory safety boundaries, error handling, benchmarks, and diffs were manually reviewed, inspected, and verified by me.
• All test suites, clippy runs (with warnings denied), benchmarks, and formatting checks were executed and validated locally prior to submission.

---

• I checked for similar changes and make sure to reference them
• I included a changelog entry for relevant new features or changes

[LucaCappelletti94]

@kishansaaai could you please add a criterion-based benchmark showing how main and this branch scale with size and depth?

[kishansaaai]

@LucaCappelletti94 Thanks for the suggestion! I've added a dedicated Criterion benchmark suite in diesel/benches/sqlite_jsonb.rs and benchmarked both main and this branch across multiple depths and sizes.

Depth Scaling — Nested Arrays ([[[[... 1 ...]]]])

Depth	main (baseline)	This Branch	Speedup
10	2.05 µs	0.83 µs	2.45x faster (-59%)
50	10.62 µs	2.32 µs	4.58x faster (-78%)
100	29.51 µs	4.08 µs	7.23x faster (-86%)
250	77.54 µs	10.58 µs	7.33x faster (-86%)
500	195.70 µs	19.94 µs	9.81x faster (-90%)
1000	409.44 µs	40.11 µs	10.21x faster (-90%)
2000	924.53 µs	75.43 µs	12.26x faster (-92%)

On main, the nested buffer copying shows quadratic growth ($14.4\times$ from 10 to 100, $13.9\times$ from 100 to 1000). On this branch, scaling is strictly linear $O(d)$ — doubling the depth from 1,000 to 2,000 takes $1.88\times$ the time (40 µs to 75 µs), yielding a 12.26x speedup at depth 2,000.

Depth Scaling — Nested Objects ({"k": {"k": ...}})

Depth	main (baseline)	This Branch	Speedup
10	5.05 µs	1.92 µs	2.63x faster (-62%)
50	25.20 µs	5.96 µs	4.23x faster (-76%)
100	49.32 µs	11.14 µs	4.43x faster (-77%)
250	157.27 µs	25.45 µs	6.18x faster (-84%)
500	343.44 µs	46.56 µs	7.38x faster (-86%)
1000	707.05 µs	99.91 µs	7.08x faster (-86%)

Size Scaling — Flat Array of Objects ([{"id": i, "name": ...}])

Elements	main (baseline)	This Branch	Speedup
10	9.92 µs	5.36 µs	1.85x faster (-46%)
100	134.16 µs	30.06 µs	4.46x faster (-78%)
1000	1.83 ms	0.30 ms	6.10x faster (-84%)
5000	4.73 ms	2.06 ms	2.30x faster (-56%)

Size Scaling — Flat Array of Scalars ([0, 1, 2, ..., N])

Elements	main (baseline)	This Branch	Speedup
10	2.43 µs	2.50 µs	~1.0x
100	19.00 µs	18.14 µs	1.05x faster
1000	171.40 µs	171.25 µs	~1.0x
10000	1.78 ms	1.72 ms	1.03x faster

For flat scalar arrays where no container nesting is present, both approaches scale linearly $O(N)$ with virtually identical times, confirming zero regression on flat inputs while providing up to 12.26x speedup on nested inputs.

I have also merged the latest upstream/main to resolve the merge conflict in diesel/src/sqlite/types/json.rs and incorporated the negative integer fix.