Skip to content

Commit eb327b6

Browse files
Yicong-Huangzhengruifeng
authored andcommitted
[SPARK-46161][PS][FOLLOWUP] Validate periods type in DataFrame.diff axis=1 branch
### What changes were proposed in this pull request? This is a follow-up to [SPARK-46161](https://issues.apache.org/jira/browse/SPARK-46161), which added `axis=1` support to pandas-on-Spark `DataFrame.diff`. The new `axis=1` branch skipped the `periods` type validation that the `axis=0` path already enforces (via `Series._diff`). This PR adds the same guard at the top of the `axis=1` branch so a non-integer `periods` raises the established `TypeError` instead of a raw Python list-index error. ### Why are the changes needed? The `axis=0` path validates `periods` in `Series._diff` and raises `TypeError("periods should be an int; however, got [...]")`. The `axis=1` branch computed `prev_idx = i - periods` and then indexed `column_labels[prev_idx]`, so a non-integer `periods` leaked an unhelpful Python error rather than the documented one. For example, before this change: ```python >>> import pyspark.pandas as ps >>> psdf = ps.DataFrame({"a": [1, 2, 3], "b": [1, 2, 3], "c": [1, 2, 3]}) >>> psdf.diff(1.5, axis=1) TypeError: list indices must be integers or slices, not float ``` After this change it matches the `axis=0` behavior: ```python >>> psdf.diff(1.5, axis=1) TypeError: periods should be an int; however, got [float] ``` ### Does this PR introduce _any_ user-facing change? Yes, but only within the unreleased `axis=1` support added by SPARK-46161. Passing a non-integer `periods` with `axis=1` now raises `TypeError: periods should be an int; however, got [...]` instead of `TypeError: list indices must be integers or slices, not float`. There is no change compared to released Spark versions, and valid calls are unaffected. ### How was this patch tested? Added a negative assertion to `test_diff` in `python/pyspark/pandas/tests/computation/test_compute.py` covering `psdf.diff(1.5, axis=1)`, mirroring the existing `axis=0` check. ### Was this patch authored or co-authored using generative AI tooling? No Closes #58259 from Yicong-Huang/spark-46161-periods-type-validation. Authored-by: Yicong Huang <17627829+Yicong-Huang@users.noreply.github.com> Signed-off-by: Ruifeng Zheng <ruifengz@foxmail.com>
1 parent 204a73d commit eb327b6

2 files changed

Lines changed: 7 additions & 0 deletions

File tree

python/pyspark/pandas/frame.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5029,6 +5029,10 @@ def diff(self, periods: int = 1, axis: Axis = 0) -> "DataFrame":
50295029
if axis == 0:
50305030
return self._apply_series_op(lambda psser: psser._diff(periods), should_resolve=True)
50315031
else:
5032+
if not isinstance(periods, int):
5033+
raise TypeError(
5034+
"periods should be an int; however, got [%s]" % type(periods).__name__
5035+
)
50325036
column_labels = self._internal.column_labels
50335037
data_col_names = self._internal.data_spark_column_names
50345038
new_columns: list[PySparkColumn] = []

python/pyspark/pandas/tests/computation/test_compute.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -204,6 +204,9 @@ def test_diff(self):
204204
self.assert_eq(pdf.diff(periods=2, axis=1), psdf.diff(periods=2, axis=1))
205205
self.assert_eq(pdf.diff(periods=-1, axis=1), psdf.diff(periods=-1, axis=1))
206206

207+
with self.assertRaisesRegex(TypeError, msg):
208+
psdf.diff(1.5, axis=1)
209+
207210
# multi-index columns
208211
columns = pd.MultiIndex.from_tuples([("x", "Col1"), ("x", "Col2"), ("y", "Col3")])
209212
pdf.columns = columns

0 commit comments

Comments
 (0)