Commit eb327b6
[SPARK-46161][PS][FOLLOWUP] Validate periods type in DataFrame.diff axis=1 branch
### What changes were proposed in this pull request?
This is a follow-up to [SPARK-46161](https://issues.apache.org/jira/browse/SPARK-46161), which added `axis=1` support to pandas-on-Spark `DataFrame.diff`. The new `axis=1` branch skipped the `periods` type validation that the `axis=0` path already enforces (via `Series._diff`). This PR adds the same guard at the top of the `axis=1` branch so a non-integer `periods` raises the established `TypeError` instead of a raw Python list-index error.
### Why are the changes needed?
The `axis=0` path validates `periods` in `Series._diff` and raises `TypeError("periods should be an int; however, got [...]")`. The `axis=1` branch computed `prev_idx = i - periods` and then indexed `column_labels[prev_idx]`, so a non-integer `periods` leaked an unhelpful Python error rather than the documented one. For example, before this change:
```python
>>> import pyspark.pandas as ps
>>> psdf = ps.DataFrame({"a": [1, 2, 3], "b": [1, 2, 3], "c": [1, 2, 3]})
>>> psdf.diff(1.5, axis=1)
TypeError: list indices must be integers or slices, not float
```
After this change it matches the `axis=0` behavior:
```python
>>> psdf.diff(1.5, axis=1)
TypeError: periods should be an int; however, got [float]
```
### Does this PR introduce _any_ user-facing change?
Yes, but only within the unreleased `axis=1` support added by SPARK-46161. Passing a non-integer `periods` with `axis=1` now raises `TypeError: periods should be an int; however, got [...]` instead of `TypeError: list indices must be integers or slices, not float`. There is no change compared to released Spark versions, and valid calls are unaffected.
### How was this patch tested?
Added a negative assertion to `test_diff` in `python/pyspark/pandas/tests/computation/test_compute.py` covering `psdf.diff(1.5, axis=1)`, mirroring the existing `axis=0` check.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #58259 from Yicong-Huang/spark-46161-periods-type-validation.
Authored-by: Yicong Huang <17627829+Yicong-Huang@users.noreply.github.com>
Signed-off-by: Ruifeng Zheng <ruifengz@foxmail.com>1 parent 204a73d commit eb327b6
2 files changed
Lines changed: 7 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5029 | 5029 | | |
5030 | 5030 | | |
5031 | 5031 | | |
| 5032 | + | |
| 5033 | + | |
| 5034 | + | |
| 5035 | + | |
5032 | 5036 | | |
5033 | 5037 | | |
5034 | 5038 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
204 | 204 | | |
205 | 205 | | |
206 | 206 | | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
207 | 210 | | |
208 | 211 | | |
209 | 212 | | |
| |||
0 commit comments