fix(go): error on NUMBER values that overflow int64 with use_high_precision=false#161
Open
zeroshade wants to merge 1 commit into
Open
fix(go): error on NUMBER values that overflow int64 with use_high_precision=false#161zeroshade wants to merge 1 commit into
zeroshade wants to merge 1 commit into
Conversation
…cision=false With use_high_precision=false, Snowflake NUMBER(n,0) columns are returned as int64. High-precision NUMBER arrives as Decimal128, and the previous compute.UnsafeCastOptions cast silently discarded the upper bits for values that do not fit in int64, returning a corrupted low-64-bit result. Convert scale-0 decimal columns with an explicit inclusive int64 bounds check and return a StatusInvalidData error when a value is out of range instead of truncating it. The bounds are checked manually because arrow-go's safe decimal->int cast rejects math.MaxInt64 itself. Also preserve a transformer's adbc.Error status code through the record transform loop so the error surfaces as InvalidData rather than Internal. Closes #129
lidavidm
reviewed
Jun 21, 2026
Comment on lines
+578
to
+580
| // The range is checked manually rather than with compute.SafeCastOptions because | ||
| // arrow-go's safe decimal->int kernel tests `value >= MaxInt64` and so rejects | ||
| // math.MaxInt64 itself; a manual check accepts both int64 bounds inclusively. |
Contributor
There was a problem hiding this comment.
Er, should we fix this upstream?
Contributor
Author
There was a problem hiding this comment.
good point. don't know why i didn't just do that when i saw this...
zeroshade
added a commit
to apache/arrow-go
that referenced
this pull request
Jun 23, 2026
…#862) ### Rationale for this change The safe `decimal128`/`decimal256` → integer cast kernel range-checks each value against the output type's bounds before truncating to the low 64 bits. The upper-bound check used `value >= max`, where `max` is the *inclusive* maximum of the output integer type (e.g. `math.MaxInt64` for `int64`, computed via `MaxOf[T]()`). As a result a value exactly equal to the type maximum was incorrectly rejected as "integer value out of bounds", even though it is a valid in-range value. This affects every integer output type (`int8`/`int16`/`int32`/`int64` and the unsigned variants), not just `int64`. This was noticed while working around the behavior downstream in the ADBC Snowflake driver (adbc-drivers/snowflake#161), where it was suggested to fix it upstream here. ### What changes are included in this PR? - In `decimalToIntImpl` (`arrow/compute/internal/kernels/numeric_cast.go`), change the upper-bound overflow check from `value >= max` to `value > max` so the inclusive maximum is accepted, matching the already-inclusive lower-bound check (`value < min`). The `decimal[T]` helper interface now requires `Greater` instead of `GreaterEqual`; both `decimal128.Num` and `decimal256.Num` already implement it. ### Are these changes tested? Yes. Added an `int64 bounds inclusive` regression subtest to `TestDecimal128ToInt` and `TestDecimal256ToInt` that assert, with `AllowIntOverflow=false`: - `math.MaxInt64` (`9223372036854775807`) and `math.MinInt64` (`-9223372036854775808`) cast successfully, and - values one beyond either bound still fail as overflow. `go test ./arrow/compute/...` passes and `go vet` is clean. ### Are there any user-facing changes? Yes — a bug fix. A safe (non-`AllowIntOverflow`) decimal→integer cast of a value equal to the destination type's maximum (e.g. `math.MaxInt64`) now succeeds instead of returning an `ErrInvalid` "integer value out of bounds" error. Values genuinely outside the range still error as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
With
use_high_precision=false, SnowflakeNUMBER(n,0)columns are returned as Arrowint64. Because Snowflake stores integer types asNUMBER(38,0)(decimal128-equivalent), high-precision columns arrive as aDecimal128array, and the previouscompute.UnsafeCastOptionscast silently discarded the upper bits for values that do not fit in int64 — returning a corrupted low-64-bit result with no error or warning.Closes #129.
Changes
record_reader.go— scale-0Decimal128/Decimal256columns are now converted via a newdecimalScale0ToInt64transformer that range-checks each value and returns anadbc.StatusInvalidDataerror when it falls outside[MinInt64, MaxInt64], instead of truncating. The bounds are checked manually (not viacompute.SafeCastOptions) because arrow-go's safe decimal→int kernel testsvalue >= MaxInt64and therefore rejectsMaxInt64itself.record_reader.go— preserve a transformer'sadbc.Errorstatus code through the record-transform loop, so the overflow surfaces asInvalidDatarather than being re-wrapped asInternal.TestDecimalScale0ToInt64unit test covering in-range values, the int64 min/max boundaries, and overflow; updatedTestIntDecimalLowPrecisionto expect aStatusInvalidDatareader error for out-of-rangeNUMBER(p,0)values (precision ≥ 19).numeric_low_precisioncaveat to document the new error-on-overflow behavior (it previously documented the silent truncation).Behavior change
Queries that previously returned a silently-truncated value for out-of-range
NUMBERnow return an error. Users who need the full value can setuse_high_precision=trueto read the column losslessly asdecimal128.Testing
go test ./...passes (unit tests; Snowflake integration tests skip without credentials).go vet ./...clean.