Skip to content

Commit af9ac19

Browse files
authored
Add adjust_ordering utility (#22628)
- Part of #22128 - ~Depends on rapidsai/rapidsmpf#1039 - Uses `SparseAlltoAll` to enforce a new `Ordering` for a channel that is already partitioned with a compatible `Ordering` - The utility will typically be used to adjust boundaries or convert from non-strict to strict boundaries. This utility is intended for operators that can exploit existing ordered/range-partitioned input, but require a different concrete boundary layout before they can do so safely. For example, a downstream operator may need to adjust from one strict `Ordering` to another with different boundaries, or convert metadata that is ordered but non-strict into strict output partitions before using chunkwise execution. The immediate motivation is ordered join/groupby/sort planning in cudf-polars: once a stream is known to be ordered, we want to repartition only the boundary-overlap regions needed to align with the target operator, rather than falling back to a full shuffle or sort. **NOTE**: "Strict" partitioning means that a unique value may only exist in one chunk. It is possible for the data to be ordered without "strict" partitioning, but we need to enforce strictness before doing a sort-based join or groupby. ### Simple example: aligning ordered join inputs Suppose two input streams are both ordered on the join keys, but their partition boundaries do not line up. A chunkwise join can only be used safely when corresponding output partitions cover the same key ranges. `adjust_ordering` provides the data-movement primitive for reshaping one ordered stream to match the other stream's strict boundaries, moving only the boundary-overlap pieces that need to change ranks. This lets a downstream join operate partition-by-partition without requiring a full hash shuffle or global sort. The same primitive can also be used by future groupby/sort optimizations that need to turn ordered-but-misaligned or non-strict partitioning into "strict" operator-ready partitions. Authors: - Richard (Rick) Zamora (https://github.com/rjzamora) Approvers: - Tom Augspurger (https://github.com/TomAugspurger) URL: #22628
1 parent 3227419 commit af9ac19

3 files changed

Lines changed: 1286 additions & 9 deletions

File tree

0 commit comments

Comments
 (0)