Skip to content

String bounds pruning: truncated bounds are valid outer bounds - #35

Merged
platypii merged 1 commit into
masterfrom
fix/string-bounds-pruning
Aug 12, 2026
Merged

String bounds pruning: truncated bounds are valid outer bounds#35
platypii merged 1 commit into
masterfrom
fix/string-bounds-pruning

Conversation

@philcunliffe

Copy link
Copy Markdown
Contributor

Fixes #34.

What

  1. isOrderableForBounds admits the byte-ordered family (string, binary, uuid, fixed[...]), so boundsMightMatch can prune files on their manifest bounds. The truncation argument: Iceberg's truncate(16) metrics keep bounds valid as OUTER bounds (a truncated lower bound is a prefix of the minimum, byte-order <= min; a truncated upper bound has its last unit incremented, >= max, or is omitted when incrementing overflows, which already reads as open-above). Every skip in boundsOpMightMatch requires the predicate to be provably outside [lo, hi], which outer bounds only ever widen, so the existing operator logic is truncation-safe as written. $ne/$nin, the only operators that would need exact single-value bounds, already keep.
  2. Strings compare by code point (compareStringsCodePoint in serde.js), which equals UTF-8 byte order - Iceberg's canonical string order for bounds and sort keys. JS < compares UTF-16 code units, which disagree whenever a character in U+E000..U+FFFF meets a supplementary character. Because compare is shared, this also fixes two latent writer-side bugs for such strings: computeColumnStats could select the wrong min/max (producing bounds that do not bound), and write-path sorting ordered them off-spec. The comparator walks code points with no allocation (the writer compares full column values per row, which can be megabytes) and throws on non-string input so safeCompare degrades a mismatched literal to keep-the-file instead of ordering garbage.
  3. Identity string partitions gain range pruning through the same comparator (compareOrder now orders strings), with identity's exact semantics: $gt 'a' on a file whose partition value is 'a' proves no row can match.

Why

Measured production impact in hyparam/hypaware-server#317: a multi-GB table whose day idiom is a string date column read its ENTIRE archive on every query because string bounds never pruned - ~8s and 2,587 MB per query against 0.74s and 12 MB for an equivalent int predicate through the same path. The manifests already carried exact bounds (lower = upper = "2026-07-09").

Tests

  • prune.bounds.test.js: new string-bounds suite - eq/range/in skip-vs-keep at boundaries, $ne keeps on single-valued bounds, truncated bounds acting as outer bounds (prefix lower, incremented upper), the UTF-16 vs UTF-8 divergence case (U+10000 vs a U+E000 bound), mismatched literal type keeps, binary bytewise pruning. The old "string range never prunes" conservative case is superseded by this suite.
  • prune.test.js: identity string partition now asserts range pruning with exact identity semantics.
  • Full suite: 639 passed. Lint and build:types clean.

Caveat noted for review

Bounds decode via TextDecoder before comparison. A writer that truncated mid-character (spec forbids this; bounds must remain valid UTF-8) would decode partial bytes to U+FFFD and could compare differently from the raw bytes. Spec-compliant writers, including this repo's truncateLower/truncateUpper, are unaffected.

🤖 Generated with Claude Code

Admit the byte-ordered family (string/binary/uuid/fixed) to bounds
pruning. Iceberg metric truncation preserves outer-bound validity: a
truncated lower bound is a prefix of the minimum, a truncated upper
bound has its last unit incremented or is omitted, so every skip that
requires the predicate to be provably outside [lo, hi] stays safe.
$ne/$nin remain unprunable.

Strings now compare by code point (compareStringsCodePoint), which is
UTF-8 byte order - Iceberg's order for bounds and sort keys. JS < uses
UTF-16 code units, which disagree when U+E000..U+FFFF meets a
supplementary character; this also fixes writer min/max selection and
write-path sort for such strings. Identity string partitions gain
range pruning through the same comparator.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@philcunliffe
philcunliffe requested a review from platypii August 12, 2026 00:38
@platypii
platypii merged commit eda5205 into master Aug 12, 2026
6 checks passed
@platypii
platypii deleted the fix/string-bounds-pruning branch August 12, 2026 06:29
platypii pushed a commit to hyparam/hypaware that referenced this pull request Aug 12, 2026
icebird 0.8.18 (hyparam/icebird#35) admits string/binary/uuid/fixed
columns to Iceberg manifest bounds pruning. Truncation keeps those
bounds valid as outer bounds, so a skip that requires the predicate to
be provably outside [lo, hi] stays sound; $ne/$nin still never prune.

This matters here twice. Locally, a date-scoped query over recorded
cache tables can now skip whole files instead of opening every one. In
hypaware-server's deployed image the two packages install separate
trees, so the cache tier resolves THIS pin while the archive tier
resolves the server's - the server bumped in hyparam/hypaware-server#326
and the cache tier stayed on 0.8.17 until this.

The same release also orders strings by code point rather than UTF-16
code units, which is the order Iceberg specifies for bounds and sort
keys. That corrects min/max bound selection and write-path sorting for
strings mixing U+E000..U+FFFF with supplementary characters; ASCII and
common text are unaffected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

String (and byte-family) columns are never bounds-pruned, but truncated bounds are still valid outer bounds

2 participants