String bounds pruning: truncated bounds are valid outer bounds - #35
Merged
Conversation
Admit the byte-ordered family (string/binary/uuid/fixed) to bounds pruning. Iceberg metric truncation preserves outer-bound validity: a truncated lower bound is a prefix of the minimum, a truncated upper bound has its last unit incremented or is omitted, so every skip that requires the predicate to be provably outside [lo, hi] stays safe. $ne/$nin remain unprunable. Strings now compare by code point (compareStringsCodePoint), which is UTF-8 byte order - Iceberg's order for bounds and sort keys. JS < uses UTF-16 code units, which disagree when U+E000..U+FFFF meets a supplementary character; this also fixes writer min/max selection and write-path sort for such strings. Identity string partitions gain range pruning through the same comparator. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
platypii
approved these changes
Aug 12, 2026
platypii
pushed a commit
to hyparam/hypaware
that referenced
this pull request
Aug 12, 2026
icebird 0.8.18 (hyparam/icebird#35) admits string/binary/uuid/fixed columns to Iceberg manifest bounds pruning. Truncation keeps those bounds valid as outer bounds, so a skip that requires the predicate to be provably outside [lo, hi] stays sound; $ne/$nin still never prune. This matters here twice. Locally, a date-scoped query over recorded cache tables can now skip whole files instead of opening every one. In hypaware-server's deployed image the two packages install separate trees, so the cache tier resolves THIS pin while the archive tier resolves the server's - the server bumped in hyparam/hypaware-server#326 and the cache tier stayed on 0.8.17 until this. The same release also orders strings by code point rather than UTF-16 code units, which is the order Iceberg specifies for bounds and sort keys. That corrects min/max bound selection and write-path sorting for strings mixing U+E000..U+FFFF with supplementary characters; ASCII and common text are unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #34.
What
isOrderableForBoundsadmits the byte-ordered family (string,binary,uuid,fixed[...]), soboundsMightMatchcan prune files on their manifest bounds. The truncation argument: Iceberg'struncate(16)metrics keep bounds valid as OUTER bounds (a truncated lower bound is a prefix of the minimum, byte-order<= min; a truncated upper bound has its last unit incremented,>= max, or is omitted when incrementing overflows, which already reads as open-above). Every skip inboundsOpMightMatchrequires the predicate to be provably outside[lo, hi], which outer bounds only ever widen, so the existing operator logic is truncation-safe as written.$ne/$nin, the only operators that would need exact single-value bounds, already keep.compareStringsCodePointin serde.js), which equals UTF-8 byte order - Iceberg's canonical string order for bounds and sort keys. JS<compares UTF-16 code units, which disagree whenever a character in U+E000..U+FFFF meets a supplementary character. Becausecompareis shared, this also fixes two latent writer-side bugs for such strings:computeColumnStatscould select the wrong min/max (producing bounds that do not bound), and write-path sorting ordered them off-spec. The comparator walks code points with no allocation (the writer compares full column values per row, which can be megabytes) and throws on non-string input sosafeComparedegrades a mismatched literal to keep-the-file instead of ordering garbage.compareOrdernow orders strings), with identity's exact semantics:$gt 'a'on a file whose partition value is'a'proves no row can match.Why
Measured production impact in hyparam/hypaware-server#317: a multi-GB table whose day idiom is a string
datecolumn read its ENTIRE archive on every query because string bounds never pruned - ~8s and 2,587 MB per query against 0.74s and 12 MB for an equivalent int predicate through the same path. The manifests already carried exact bounds (lower = upper = "2026-07-09").Tests
prune.bounds.test.js: new string-bounds suite - eq/range/in skip-vs-keep at boundaries,$nekeeps on single-valued bounds, truncated bounds acting as outer bounds (prefix lower, incremented upper), the UTF-16 vs UTF-8 divergence case (U+10000vs aU+E000bound), mismatched literal type keeps, binary bytewise pruning. The old "string range never prunes" conservative case is superseded by this suite.prune.test.js: identity string partition now asserts range pruning with exact identity semantics.build:typesclean.Caveat noted for review
Bounds decode via
TextDecoderbefore comparison. A writer that truncated mid-character (spec forbids this; bounds must remain valid UTF-8) would decode partial bytes to U+FFFD and could compare differently from the raw bytes. Spec-compliant writers, including this repo'struncateLower/truncateUpper, are unaffected.🤖 Generated with Claude Code