buffa in production: full prost->buffa migration (+21.65%) — memory & binary-size tradeoffs worth investigating

[jlucaso1]

Hey — the full prost 0.13 -> buffa 0.8.1 migration I've been chipping away at is now landing in production (a large WhatsApp protocol implementation), so I wanted to share the end-to-end results as a real-world "buffa in production" data point, and flag two tradeoffs that look like they might be addressable in buffa itself rather than at the app level.

Overall it's a clear win — the full CodSpeed suite came out +21.65% net across 179 benchmarks (17 improvements, 6 regressions, 156 unchanged), on top of the struct-size and decode-speed wins from the earlier issues. Sharing the regressions too so the tradeoff is on the table and this isn't one-sided.

PR (real production migration, reproducible CodSpeed run attached to it): oxidezap/whatsapp-rust#557

The wins (so it's not one-sided)

A sample of the big movers vs prost on the same suite:

benchmark	metric	prost	buffa	Δ
decode_plaintext[text_reply]	memory	3.4 KB	1.3 KB	×2.6
decode_plaintext[group_skdm_text]	time	8.4 µs	6.0 µs	+40%
history_sync_stream_drain	memory	431 KB	243 KB	+77%
process_patch_50_validated	memory	171 KB	48 KB	×3.6
dm_session_establishment	memory	3.7 KB	2.4 KB	+55%
full_dm_conversation	memory	9.8 KB	7.8 KB	+27%
decode_record[contact]	time	44.6 µs	35.2 µs	+27%

Decode across the board is both faster and lighter, and the deep-tree history-sync/appstate paths improved the most — exactly what the boxed-field + tighter-codegen work was aiming at.

Tradeoff 1: a subset of decode/encode shapes regressed on memory

A handful of benchmarks allocate more under buffa than prost. Two shapes stand out:

benchmark	metric	prost	buffa	Δ
decode_record[contact]	memory	385 B	1,113 B	−65%
decode_record[star]	memory	484 B	1,085 B	−55%
group_send_skdm_256	memory	415.5 KB	461.8 KB	−10%
group_send_skdm_50	memory	73.4 KB	82.5 KB	−11%

The decode_record case is the interesting one: memory ~3x but time −25% (it's in the wins table above), so it's a clean time/memory trade rather than a straight loss — buffa is doing fewer passes but allocating more per decode. The group-send fanout cases allocate ~10% more per encoded recipient.

I haven't pinned the exact source, but the candidates all live in buffa: eager materialization of small nested/repeated fields during decode, per-field MessageField box allocations, or SizeCache spill growth. Worth a look at whether the extra allocation on these small-message decode paths is inherent to buffa's representation or is reclaimable — the time is already better, so closing the memory gap would make these strict wins.

Tradeoff 2: final binary grew ~+3.85% despite codegen shrinking 23%

This is the one I'd most like a second opinion on, because the two halves point in opposite directions:

Metric                              prost        buffa        Δ
bin size (stripped)                 10.20 MiB    10.59 MiB    +401.91 KiB (+3.85%)
bin .text                            8.24 MiB     8.63 MiB    +397.62 KiB (+4.71%)
llvm-lines  wacore (proto crate)     649,184      497,172     -152,012 (-23.42%)
llvm-lines  wacore copies             17,998       17,033       -965 (-5.36%)
llvm-lines  whatsapp-rust lib        678,181      710,609     +32,428 (+4.78%)
llvm-lines  whatsapp-rust lib copies  21,000       23,096      +2,096 (+9.98%)

The generated proto crate got 23% leaner in llvm-lines (buffa's codegen is tighter than prost's) — but the final binary still grew ~400 KiB, and the growth tracks the consuming crate: whatsapp-rust lib llvm-lines +4.78% and, tellingly, monomorphization copies +9.98%. So the extra .text isn't in the generated code — it's in more monomorphized instances downstream.

My working hypothesis is that buffa's encode/decode entry points are generic over the buffer (impl Buf / impl BufMut) and the message/field wrappers are generic (MessageField<T>, EnumValue<E>, the view types), so each concrete message type instantiates those generics in the consuming crate — whereas prost's hot paths are more concrete (bytes::Bytes in fixed positions). With this repo's ~hundreds of message types, that multiplies. But that's a guess — the copies +9.98% number is the concrete signal and I'd rather hear from someone who knows the codegen where those instances are coming from.

Open questions / directions (leaving the fix open):

• Could the per-message encode/decode public entry points take a concrete buffer type (or &mut dyn Buf) at the boundary and only go generic internally, so N message types don't each monomorphize the buffer generic?
• Is the +4.78% consumer-side growth mostly the MessageField/EnumValue/view generic instantiations, or the per-message compute_size/write_to/merge_field bodies?
• Is any of this a codegen #[inline] policy that could be relaxed to dedup across message types?

Repro is the PR itself (the Binary-Size and CodSpeed jobs run on every push, numbers above are from that CI); happy to pull any specific cargo llvm-lines / cargo bloat breakdown from the tree if it helps localize where the copies land.

[jlucaso1]

Follow-up: I root-caused the regressions from the report above with local callgrind/cachegrind A/B against the prost build (per-iteration deltas between two iteration counts, so the harness cancels out) plus dhat for the allocation ones. Three findings worth sharing, one with a measured prototype fix.

1. The "time" regressions are a cold-cache measurement artifact — with a real mechanism behind it

bench_message_encoding_simple (-14.8%) and bench_content_extraction_simple (-11.9%) do not reproduce as instruction-count or hot-cache regressions. Locally, buffa beats prost on every axis for the same bench:

per iteration (cachegrind, N=201k minus N=1k)	prost	buffa 0.8.1
I refs	~+26% more	baseline
D refs	1051	876
LL refs (whole run, hot)	493K	457K

The CodSpeed per-component breakdown shows the entire regression sitting in memoryAccess, and the flamegraph attributes it to the per-field presence checks with self time ~92% "memory". The mechanism: generated compute_size/write_to do if self.field.is_set() { self.field.compute_size(...) }, and the method call resolves through MessageField's Deref, whose None arm references T::default_instance() — a per-type OnceBox static. A wide message walk therefore embeds potential references to ~70 distinct per-type statics scattered across the binary. On a short, cold-cache run (which is what an instrumented benchmark iteration is), those are first-touch accesses charged at RAM cost; in any steady-state process they are hot and cost nothing. So: real mechanism, but only visible in cold-run cost models — production profiles on our side show the encode path at <1% CPU.

2. Prototype codegen fix, with an honest trade-off

I patched impl_message.rs (TYPE_MESSAGE/TYPE_GROUP arms in both size and write paths) to bind once instead of checking twice:

if let Some(__value) = self.#ident.as_option() {
    let inner_size = __value.compute_size(__cache);
    ...
}

Semantically identical (is_set() == as_option().is_some(); deref-on-set == the Some payload). Measured on the same bench:

• eliminates the Deref/default_instance static references entirely (the cold-touch set shrinks accordingly);
• but costs ~+100 I refs and ~+100 D refs per hot iteration on this message shape (the extra binding appears to change register allocation in the ~110-field walk).

So it fixes the cold-run/benchmark behavior at a small hot-loop cost — not an unconditional win, which is why I'm reporting it rather than opening a PR. If the hot-loop delta can be avoided (e.g. as_option().unwrap_unchecked() after is_set, or keeping the check but devirtualizing the deref), it becomes strictly better.

3. The decode memory regressions are the boxed-field repr, and PointerRepr::Inline already solves it — but is not in 0.8.1

bench_decode_record[contact] (385 B -> 1,113 B) is exactly one Box<SyncActionValue> (~632 B) allocated and immediately moved out per record: prost stored that singular message field inline, buffa's default repr boxes it. dhat shows the box as the only new allocation site. The box_type_in(PointerRepr::Inline, ...) knob from #250 is precisely the fix on the consumer side — but 0.8.1 appears to have been cut before the June 27-29 merges, so Inline (and #259, #260, #265) are not in the released crate. A 0.8.2 release would let consumers resolve this class of regression with one build.rs line; happy to report back numbers once it's out.