Commit d7cb659
[SPARK-59011][PS] Fix descending rank first tie ordering
### What changes were proposed in this pull request?
This pull request fixes the tie-breaking order of pandas-on-Spark `Series.rank(method="first", ascending=False)`.
The value column follows the requested ascending direction, while the natural-order column is always ordered ascending so that equal values are ranked according to their original occurrence order.
A regression test for descending ranking with `method="first"` is added.
### Why are the changes needed?
The existing implementation applies descending order to both the value column and the natural-order tie-breaker. This reverses the original order of equal values and produces results inconsistent with pandas.
For `pd.Series([1, 2, 3, 1])`, pandas returns:
`[3.0, 2.0, 1.0, 4.0]`
The previous pandas-on-Spark result was:
`[4.0, 2.0, 1.0, 3.0]`
### Does this PR introduce _any_ user-facing change?
Yes. `Series.rank(method="first", ascending=False)` now preserves the original occurrence order when breaking ties, matching pandas behavior.
### How was this patch tested?
A regression assertion was added to `SeriesStatTests.test_rank`.
The test failed before the fix with a 50% value difference and passed after the fix:
./python/run-tests --python-executables python3 \
--testnames 'pyspark.pandas.tests.series.test_stat SeriesStatTests.test_rank' \
-p 1
The Spark assembly build, Ruff checks, Ruff formatting check, and `git diff --check` also passed.
This contribution was prepared and reviewed by our university project team, including Phan Trung Nhut and Bui Nam Viet. We license this contribution to the project under the Apache License, Version 2.0.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
Closes #58337 from namviet157/fix-SPARK-59011.
Lead-authored-by: NaVis <92081226+namviet157@users.noreply.github.com>
Co-authored-by: Phan Trung Nhựt <173559382+nhutphansayhi@users.noreply.github.com>
Signed-off-by: Yicong-Huang <17627829+Yicong-Huang@users.noreply.github.com>1 parent 2a7cfea commit d7cb659
2 files changed
Lines changed: 5 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4321 | 4321 | | |
4322 | 4322 | | |
4323 | 4323 | | |
4324 | | - | |
| 4324 | + | |
4325 | 4325 | | |
4326 | 4326 | | |
4327 | 4327 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
319 | 319 | | |
320 | 320 | | |
321 | 321 | | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
322 | 326 | | |
323 | 327 | | |
324 | 328 | | |
| |||
0 commit comments