Dedup-first eager aggregation for GROUP BY over string functions of one column

[bgmcmullen]

The query that was breaking

hypaware-server's 2026-08-26 outage query (hyparam/hypaware-server#394 and #395 hold the containment side) still OOMs a 4GB heap on v0.16.3:

SELECT substr(regexp_replace(system_text, '\s+', ' '), 1, 150) AS k, COUNT(*) AS n
FROM ai_gateway_messages GROUP BY k ORDER BY n DESC

Data: 94,392 rows sharing 3,445 distinct ~90KB system_text values by reference (8.2GB logical, parquet dictionary decode).

The deciding characteristic is a fat intermediate hidden behind a small final key. #60 sizes chunks from the evaluated key bytes, and substr(..., 1, 150) makes every key look 150 bytes wide, so the chunk grows back to its 4000-row cap while each row still materializes a full ~90KB regexp_replace copy inside the expression, where the sizing never sees it. Any GROUP BY substr(lower(col), ...) or GROUP BY substr(CAST(col AS VARCHAR), ...) fails the same way. Unwrapped (GROUP BY regexp_replace(...), #60's own bench shape) stays bounded, so the wrapper is what breaks it.

The fix: dedup first, transform after

HashAggregate(groupBy: [f(col)], columns: [key, COUNT(*)], child)
  becomes
HashAggregate(groupBy: [f(col)], columns: [key, SUM(partial)],
  child: HashAggregate(groupBy: [col], columns: [col, COUNT(*) AS partial], child))

Grouping on the raw column only compares references, so the inner stage collapses 94k rows to 3,445 before f runs once per survivor. Answer-preserving because f is deterministic, and the executor is untouched: both stages are ordinary HashAggregate nodes, still covered by #60's chunking. This is the relational form of the "dictionary-aware batch evaluation" #60's notes name as a follow-up, and it covers the repeated-input case #57/#58 targeted without a result cache. New file src/plan/eagerAggregate.js, applied where planSelect builds the aggregate node.

When it fires

All of: one GROUP BY key over exactly one unqualified column through deterministic scalar constructs; the key holds at least one string-producing function or string cast; every output is the key or a splittable aggregate (COUNT/COUNTIF/SUM/MIN/MAX, no DISTINCT/FILTER); no HAVING; and every aggregate-level ORDER BY term matches an output column, so it can be redirected to that alias (re-evaluating COUNT(*) on the merge stage would count partial rows). Otherwise the node is returned unchanged.

Results

One process per shape, executeSql directly, 4GB cap, peak heapUsed sampled at 25ms (peaks are lower bounds; pass/fail is the reliable signal). Top two rows run 3x.

Query	Deciding characteristic	v0.16.3	This PR
GROUP BY substr(regexp_replace(text,…),1,150)	fat intermediate behind a small key; values repeat	OOM, 3/3	14.6 / 15.3 / 14.8s, 761MB, 3/3
GROUP BY regexp_replace(text,…)	fat value is the key; values repeat	171s, 1.56GB	10.4s, 16x faster
GROUP BY text, GROUP BY id % 10, JSON_EXTRACT(...), LIKE / REGEXP_LIKE	no string function over the key, or no aggregate at all	ok	unchanged, declines
SELECT regexp_replace(text,…) FROM t	no aggregate: the ~8.2GB result is the answer	OOM	OOM
SELECT upper(text) AS k FROM t ORDER BY k (no LIMIT)	a full sort retains every key by definition	OOM	OOM
wrapped key over 60k all-distinct ~90KB values	nothing repeats, so there is no dedup to win	OOM	OOM

Rows 4-5 need bounded sort-key evaluation and a caller-side refusal, not a plan rewrite (ORDER BY ... LIMIT n is already bounded by the existing top-k path). Row 6 is not a regression - v0.16.3 fails it too, since the sizing blindness is independent of cardinality - but the mechanism does shift from evaluation churn to dedup-map retention.

I built and measured a byte-capped flushing dedup map for row 6 and rejected it: on dictionary data it ran 136s against 14.8s, because 3,445 x 90KB is ~620MB of key bytes, far above any sane bound, so the map flushes nearly every chunk and the merge stage re-runs f on ~82k rows instead of 3,445. It still OOM'd on all-distinct data. A byte tally cannot tell a shared dictionary reference (~8 bytes of real cost) from a uniquely-held one (~180KB), so row 6 wants a cardinality signal from the data source rather than an executor heuristic.

Tests

16 in test/plan/plan.eagerAggregate.test.js: rewrite shape (incident query, string casts, ORDER BY redirection), all nine decline conditions, and execution equivalence over repeated values, distinct values normalizing to one key, NULL keys, and NULL-skipping COUNT (COUNT/SUM/MIN/MAX, CTE, alias ORDER BY). Suite 2,009 green; tsc and eslint clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01L7LMSa2odTyibuvg4AzdGg