Commit 1094e68
[SPARK-58821][SQL] Return the recomputed length instead of an internal error in Sequence
### What changes were proposed in this pull request?
`Sequence.sequenceLength` computes the sequence length in `Long` and falls back to `BigInt` when that arithmetic overflows. The fallback recomputes the length exactly, raises `COLLECTION_SIZE_LIMIT_EXCEEDED` if it is too large to allocate, and then throws `internalError("Unreachable code reached.")` on the assumption that no other outcome is possible.
That assumption does not hold. `Math.subtractExact(stop, start)` overflows whenever `stop - start` exceeds the `Long` range, which says nothing about the length, because the length also depends on `step`. For a large step the exact length is small, no limit error is raised, and control reaches the `internalError` with the correct length sitting unused in `safeLen`.
This PR returns `safeLen.toInt` instead of throwing. The check immediately above already bounds `safeLen` by `ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH`, and the caller has already rejected boundaries whose step points the wrong way, so the value is positive and the narrowing conversion is exact. The now-unused `SparkException.internalError` import is removed.
No other behaviour changes: an overflow with a length that really is too large still raises `COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER` from the same fallback.
### Why are the changes needed?
`sequence()` over BIGINT raises an `INTERNAL_ERROR` (SQLSTATE XX000) for queries that have a small, well-defined result. `INTERNAL_ERROR` is reserved for conditions that indicate a bug in Spark, so surfacing it for ordinary user input is wrong on both counts: the query should have succeeded, and the error tells the user to report a bug rather than fix their query.
```sql
SELECT sequence(-9223372036854775808L, 9223372036854775807L, 9223372036854775807L);
-- [INTERNAL_ERROR] Unreachable code reached. SQLSTATE: XX000
```
The element-filling code already handles steps of this magnitude correctly, so only the length computation was at fault: `sequence(-4611686018427387904L, 4611686018427387903L, 4611686018427387903L)`, whose endpoints are close enough together to avoid the overflow, returns `[-4611686018427387904, -1, 4611686018427387902]` on master today.
`sequenceLength` was introduced in this shape by SPARK-43393 (`afc4c49927cb`), which fixed a real correctness bug where the `Long` length computation overflowed silently and `sequence` returned an empty array. The overflow cases that fix was written against all had huge lengths, which is why the tail looked unreachable. The method has not changed since, so every release containing SPARK-43393 is affected.
### Does this PR introduce _any_ user-facing change?
Yes. Queries that failed with `INTERNAL_ERROR` now return their result. Both the interpreted and the codegen path are affected, since both call `Sequence.sequenceLength`.
Before:
```sql
SELECT sequence(-9223372036854775808L, 9223372036854775807L, 9223372036854775807L);
-- [INTERNAL_ERROR] Unreachable code reached. SQLSTATE: XX000
SELECT sequence(-9223372036854775808L, 0L, 4611686018427387904L);
-- [INTERNAL_ERROR] Unreachable code reached. SQLSTATE: XX000
```
After:
```sql
SELECT sequence(-9223372036854775808L, 9223372036854775807L, 9223372036854775807L);
-- [-9223372036854775808, -1, 9223372036854775806]
SELECT sequence(-9223372036854775808L, 0L, 4611686018427387904L);
-- [-9223372036854775808, -4611686018427387904, 0]
```
No query that previously succeeded changes its result, and no query that previously raised `COLLECTION_SIZE_LIMIT_EXCEEDED` stops raising it. Since the previous behaviour was an internal error, no migration guide entry is needed.
### How was this patch tested?
Three cases added to the `Sequence of numbers` test in `CollectionExpressionsSuite`, next to the existing SPARK-43393 overflow cases. `checkEvaluation` exercises both the interpreted and the codegen path:
* a positive step at both `Long` extremes,
* a positive step where only `stop - start` overflows,
* the negative-step mirror.
`build/sbt 'catalyst/testOnly *CollectionExpressionsSuite'` passes (62 tests). The existing SPARK-43393 cases, which cover the fallback still raising the limit error, pass unchanged.
Also verified end to end against a local `SparkSession`, including a non-foldable form reading the arguments from a temp view, to confirm the fix applies at runtime and not only through constant folding.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
Closes #58047 from SEPURI-SAI-KRISHNA/SPARK-58821-sequence-length-internal-error.
Authored-by: Sepuri Sai Krishna <saik20533@gmail.com>
Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>
(cherry picked from commit de60883)
Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>1 parent 00d3243 commit 1094e68
2 files changed
Lines changed: 22 additions & 3 deletions
File tree
- sql/catalyst/src
- main/scala/org/apache/spark/sql/catalyst/expressions
- test/scala/org/apache/spark/sql/catalyst/expressions
Lines changed: 8 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
27 | 26 | | |
28 | 27 | | |
29 | 28 | | |
| |||
3420 | 3419 | | |
3421 | 3420 | | |
3422 | 3421 | | |
3423 | | - | |
| 3422 | + | |
| 3423 | + | |
| 3424 | + | |
| 3425 | + | |
3424 | 3426 | | |
3425 | 3427 | | |
3426 | 3428 | | |
3427 | 3429 | | |
3428 | 3430 | | |
3429 | 3431 | | |
3430 | | - | |
| 3432 | + | |
| 3433 | + | |
| 3434 | + | |
| 3435 | + | |
3431 | 3436 | | |
3432 | 3437 | | |
3433 | 3438 | | |
| |||
Lines changed: 14 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1137 | 1137 | | |
1138 | 1138 | | |
1139 | 1139 | | |
| 1140 | + | |
| 1141 | + | |
| 1142 | + | |
| 1143 | + | |
| 1144 | + | |
| 1145 | + | |
| 1146 | + | |
| 1147 | + | |
| 1148 | + | |
| 1149 | + | |
| 1150 | + | |
| 1151 | + | |
| 1152 | + | |
| 1153 | + | |
1140 | 1154 | | |
1141 | 1155 | | |
1142 | 1156 | | |
| |||
0 commit comments