Skip to content

Commit d7cb659

Browse files
namviet157trungnhutphan
authored andcommitted
[SPARK-59011][PS] Fix descending rank first tie ordering
### What changes were proposed in this pull request? This pull request fixes the tie-breaking order of pandas-on-Spark `Series.rank(method="first", ascending=False)`. The value column follows the requested ascending direction, while the natural-order column is always ordered ascending so that equal values are ranked according to their original occurrence order. A regression test for descending ranking with `method="first"` is added. ### Why are the changes needed? The existing implementation applies descending order to both the value column and the natural-order tie-breaker. This reverses the original order of equal values and produces results inconsistent with pandas. For `pd.Series([1, 2, 3, 1])`, pandas returns: `[3.0, 2.0, 1.0, 4.0]` The previous pandas-on-Spark result was: `[4.0, 2.0, 1.0, 3.0]` ### Does this PR introduce _any_ user-facing change? Yes. `Series.rank(method="first", ascending=False)` now preserves the original occurrence order when breaking ties, matching pandas behavior. ### How was this patch tested? A regression assertion was added to `SeriesStatTests.test_rank`. The test failed before the fix with a 50% value difference and passed after the fix: ./python/run-tests --python-executables python3 \ --testnames 'pyspark.pandas.tests.series.test_stat SeriesStatTests.test_rank' \ -p 1 The Spark assembly build, Ruff checks, Ruff formatting check, and `git diff --check` also passed. This contribution was prepared and reviewed by our university project team, including Phan Trung Nhut and Bui Nam Viet. We license this contribution to the project under the Apache License, Version 2.0. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) Closes #58337 from namviet157/fix-SPARK-59011. Lead-authored-by: NaVis <92081226+namviet157@users.noreply.github.com> Co-authored-by: Phan Trung Nhựt <173559382+nhutphansayhi@users.noreply.github.com> Signed-off-by: Yicong-Huang <17627829+Yicong-Huang@users.noreply.github.com>
1 parent 2a7cfea commit d7cb659

2 files changed

Lines changed: 5 additions & 1 deletion

File tree

python/pyspark/pandas/series.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4321,7 +4321,7 @@ def _rank(
43214321
window = (
43224322
Window.orderBy(
43234323
asc_func(self.spark.column),
4324-
asc_func(F.col(NATURAL_ORDER_COLUMN_NAME)),
4324+
F.col(NATURAL_ORDER_COLUMN_NAME).asc(),
43254325
)
43264326
.partitionBy(*part_cols)
43274327
.rowsBetween(Window.unboundedPreceding, Window.currentRow)

python/pyspark/pandas/tests/series/test_stat.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -319,6 +319,10 @@ def test_rank(self):
319319
self.assert_eq(pser.rank(method="min"), psser.rank(method="min").sort_index())
320320
self.assert_eq(pser.rank(method="max"), psser.rank(method="max").sort_index())
321321
self.assert_eq(pser.rank(method="first"), psser.rank(method="first").sort_index())
322+
self.assert_eq(
323+
pser.rank(method="first", ascending=False),
324+
psser.rank(method="first", ascending=False).sort_index(),
325+
)
322326
self.assert_eq(pser.rank(method="dense"), psser.rank(method="dense").sort_index())
323327

324328
non_numeric_pser = pd.Series(["a", "c", "b", "d"], name="x", index=[10, 11, 12, 13])

0 commit comments

Comments
 (0)