[PIPELINE] Add loop-carried TLE pipe cursors - #991
Draft
Prophet-Hongyi wants to merge 1 commit into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add an opt-in, first-class
pipe_cursor<capacity>that carries cyclic pipestage:i32andphase:i1as loop state.The existing endpoint API accepts an absolute iteration. For dynamic loop
iterations, the frontend immediately expands every position into
iter % capacityand(iter // capacity) % 2; pipe IR then retains only thecomputed stage and phase operands. This loses the fact that a sequential loop
can advance a ring cursor with one increment, one wrap test, and a phase toggle.
With this change, kernels may initialize
cursor = endpoint.cursor(iter)andadvance it with
cursor = cursor.advance(). Existing absolute-iteration callsremain source-compatible and unchanged.
In a matched H20 FP8 einsum experiment, the cursor form removed
214,840dynamic instructions (
8.14%) with identical launch resources and improvedfive-process CUDA Graph latency from
0.0373608 msto0.0372621 ms(
0.26%). The benchmark required a companion kernel variant that opts intothe new API; this PR does not automatically rewrite existing kernels.
Review map
pipe_cursor_type: it flattens to exactlyi32 stageandi1 phase,and includes capacity in its type identity/mangling.
pipe_cursor.advance(): increment stage, wrap to zero at capacity,and toggle phase only on wrap.
pipe_value.cursor()and_stage_phase(): cursor consumers bypassabsolute-iteration recomputation, while capacity mismatches fail before IR
construction.
language/pipe.pyandlanguage/__init__.py.i32, i1iter_args and its bodycontains pipe lifecycle operations but no
arith.remsiorarith.divsi.Supported contract
wait(iter),release(iter),acquire(iter), andcommit(iter)remain validpipe.cursor(iter),endpoint.cursor(iter), andpipe_cursor.advance()i32) plus generation phase (i1)Why this belongs at the language/type layer
The current frontend erases sequential cursor semantics before pipe lowering:
Local memoization and power-of-two strength reduction were tested separately;
both produced byte-identical final cubins because existing compiler passes
already handle those local expressions. The missing information is the
cross-iteration state transition itself.
The cursor makes that state explicit without changing the existing IR op
contract or removing random access:
Validation
Current Draft source identity:
805d3863ad04f3cd989040e59c9a54860c610137based ondf824d72097a31b4dca479ddfb7f480a470b7d66.The H20 correctness, timing, and NCU gates below ran on pre-rebase commit
c81241609ada46c75b87e54bdeae24e2eae7b799. Before publication, the singlefeature commit was replayed onto the current base and its stable patch ID
remained unchanged; the two intervening upstream commits do not touch this
PR's files. These results are patch-identical pre-rebase evidence, not an
exact-current-commit H20 rerun.
43 passed1 passedi32, i1state and no loop-bodyremsi/divsi5/5cursor outputs bitwise equal to matched Gluon referencesPerformance evidence
Baseline: the same companion typed-pipe kernel using the existing absolute
iteration endpoint API.
Evidence boundary: the patch-identical pre-rebase cursor implementation
plus a companion operator-level kernel migration on one H20 shape/config. This
is not an exact-current-commit rerun and is not evidence that unchanged kernels
improve automatically.
Setup: NVIDIA H20,
pro_b1=(b=1,h=16,r=7168,d=1024),4 warps / 8 stages / tile_order=0, five fresh processes, 30 warmups and 200CUDA Graph samples per process. Both TLE variants used the same FlagGems typed
pipe kernel; the candidate changed only the pipe position from absolute
iteration to the new cursor API. The table reports the median of per-process
average latency.
0.0373608 ms0.0372621 ms0.99736x(0.26%lower)2,639,0022,424,162-214,840(-8.14%)NCU
2025.4.1was run under authorized container root because the H20 driverrequires profiling privilege. Correctness, normal tests, and independent
timing ran as the mapped non-root user. The two profiled outputs were bitwise
equal.
The NCU replay duration is diagnostic only and is not used as the latency
claim. The
0.26%latency result comes from the separate five-process CUDAGraph gate.
Five-process CUDA Graph averages — absolute values used for the median
Files changed
python/triton/experimental/tle/language/gpu/types.py— cursor type/value, initialization, advance, and endpoint consumption.
python/triton/experimental/tle/language/pipe.py— public cursor alias.
python/triton/experimental/tle/language/__init__.py— top-level export.
python/test/tle/unit/test_tle.py— frontend/type/capacity tests.
python/test/tle/integration/test_tle_pipe_cursor.py— TTIR loop-state and arithmetic regression.
Known limits
rewrite. Existing kernels keep their current code generation until migrated.
shape, and configuration. The dynamic-instruction result is strong mechanism
evidence; the observed
0.26%latency result must not be generalized to allTLE kernels.
benchmarked. Existing absolute-iteration behavior remains the compatibility
path for those users.