Commit af9ac19
authored
Add
- Part of #22128
- ~Depends on rapidsai/rapidsmpf#1039
- Uses `SparseAlltoAll` to enforce a new `Ordering` for a channel that is already partitioned with a compatible `Ordering` - The utility will typically be used to adjust boundaries or convert from non-strict to strict boundaries.
This utility is intended for operators that can exploit existing ordered/range-partitioned input, but require a different concrete boundary layout before they can do so safely. For example, a downstream operator may need to adjust from one strict `Ordering` to another with different boundaries, or convert metadata that is ordered but non-strict into strict output partitions before using chunkwise execution.
The immediate motivation is ordered join/groupby/sort planning in cudf-polars: once a stream is known to be ordered, we want to repartition only the boundary-overlap regions needed to align with the target operator, rather than falling back to a full shuffle or sort.
**NOTE**: "Strict" partitioning means that a unique value may only exist in one chunk. It is possible for the data to be ordered without "strict" partitioning, but we need to enforce strictness before doing a sort-based join or groupby.
### Simple example: aligning ordered join inputs
Suppose two input streams are both ordered on the join keys, but their partition boundaries do not line up. A chunkwise join can only be used safely when corresponding output partitions cover the same key ranges. `adjust_ordering` provides the data-movement primitive for reshaping one ordered stream to match the other stream's strict boundaries, moving only the boundary-overlap pieces that need to change ranks.
This lets a downstream join operate partition-by-partition without requiring a full hash shuffle or global sort.
The same primitive can also be used by future groupby/sort optimizations that need to turn ordered-but-misaligned or non-strict partitioning into "strict" operator-ready partitions.
Authors:
- Richard (Rick) Zamora (https://github.com/rjzamora)
Approvers:
- Tom Augspurger (https://github.com/TomAugspurger)
URL: #22628adjust_ordering utility (#22628)1 parent 3227419 commit af9ac19
3 files changed
Lines changed: 1286 additions & 9 deletions
File tree
- python/cudf_polars
- cudf_polars/streaming/actor_graph
- collectives
- tests/streaming
0 commit comments