Skip to content

[PIPELINE] Add loop-carried TLE pipe cursors - #991

Draft
Prophet-Hongyi wants to merge 1 commit into
flagos-ai:mainfrom
Prophet-Hongyi:codex/tle-pipe-cursor
Draft

[PIPELINE] Add loop-carried TLE pipe cursors#991
Prophet-Hongyi wants to merge 1 commit into
flagos-ai:mainfrom
Prophet-Hongyi:codex/tle-pipe-cursor

Conversation

@Prophet-Hongyi

Copy link
Copy Markdown

Summary

Add an opt-in, first-class pipe_cursor<capacity> that carries cyclic pipe
stage:i32 and phase:i1 as loop state.

The existing endpoint API accepts an absolute iteration. For dynamic loop
iterations, the frontend immediately expands every position into
iter % capacity and (iter // capacity) % 2; pipe IR then retains only the
computed stage and phase operands. This loses the fact that a sequential loop
can advance a ring cursor with one increment, one wrap test, and a phase toggle.

With this change, kernels may initialize cursor = endpoint.cursor(iter) and
advance it with cursor = cursor.advance(). Existing absolute-iteration calls
remain source-compatible and unchanged.

In a matched H20 FP8 einsum experiment, the cursor form removed 214,840
dynamic instructions (8.14%) with identical launch resources and improved
five-process CUDA Graph latency from 0.0373608 ms to 0.0372621 ms
(0.26%). The benchmark required a companion kernel variant that opts into
the new API; this PR does not automatically rewrite existing kernels.

Review map

  1. Review pipe_cursor_type: it flattens to exactly i32 stage and i1 phase,
    and includes capacity in its type identity/mangling.
  2. Review pipe_cursor.advance(): increment stage, wrap to zero at capacity,
    and toggle phase only on wrap.
  3. Review pipe_value.cursor() and _stage_phase(): cursor consumers bypass
    absolute-iteration recomputation, while capacity mismatches fail before IR
    construction.
  4. Review endpoint forwarding and public exports in language/pipe.py and
    language/__init__.py.
  5. Review the TTIR regression: the loop carries i32, i1 iter_args and its body
    contains pipe lifecycle operations but no arith.remsi or arith.divsi.

Supported contract

Surface This PR
Existing API wait(iter), release(iter), acquire(iter), and commit(iter) remain valid
New API pipe.cursor(iter), endpoint.cursor(iter), and pipe_cursor.advance()
Cursor state Physical stage (i32) plus generation phase (i1)
Safety A cursor from a different pipe capacity is rejected
Activation Opt-in; a kernel must carry and pass the cursor explicitly
Lowering Existing lifecycle operations and NVWS lowering remain unchanged
Random access Preserved through the existing absolute-iteration path
Fallback/compatibility Existing absolute-iteration endpoint calls remain the default path

Why this belongs at the language/type layer

The current frontend erases sequential cursor semantics before pipe lowering:

absolute iteration
      -> frontend modulo/division
      -> pipe op(stage, phase)
      -> NVWS lowering forwards scalar operands

Local memoization and power-of-two strength reduction were tested separately;
both produced byte-identical final cubins because existing compiler passes
already handle those local expressions. The missing information is the
cross-iteration state transition itself.

The cursor makes that state explicit without changing the existing IR op
contract or removing random access:

cursor(stage, phase)
      -> scf.for iter_args(stage, phase)
      -> lifecycle op consumes state directly
      -> cursor.advance()

Validation

Current Draft source identity:
805d3863ad04f3cd989040e59c9a54860c610137 based on
df824d72097a31b4dca479ddfb7f480a470b7d66.

The H20 correctness, timing, and NCU gates below ran on pre-rebase commit
c81241609ada46c75b87e54bdeae24e2eae7b799. Before publication, the single
feature commit was replayed onto the current base and its stable patch ID
remained unchanged; the two intervening upstream commits do not touch this
PR's files. These results are patch-identical pre-rebase evidence, not an
exact-current-commit H20 rerun.

Gate Location Result What it proves
Cursor/frontend TLE unit tests NVIDIA H20 historical PASS — 43 passed Cursor typing, endpoint consumption, capacity validation, and public exports
Cursor TTIR integration test NVIDIA H20 historical PASS — 1 passed Loop-carried i32, i1 state and no loop-body remsi/divsi
Companion FP8 kernel output audit NVIDIA H20 historical PASS — 5/5 cursor outputs bitwise equal to matched Gluon references Cursor lifecycle preserves the tested kernel result
Root NCU comparison NVIDIA H20 historical PASS Dynamic instruction reduction with identical launch/resource metrics
Pre-commit hooks Local, current Draft head PASS ruff, yapf, mypy, syntax, whitespace, and repository checks passed

Performance evidence

Baseline: the same companion typed-pipe kernel using the existing absolute
iteration endpoint API.

Evidence boundary: the patch-identical pre-rebase cursor implementation
plus a companion operator-level kernel migration on one H20 shape/config. This
is not an exact-current-commit rerun and is not evidence that unchanged kernels
improve automatically.

Setup: NVIDIA H20, pro_b1=(b=1,h=16,r=7168,d=1024),
4 warps / 8 stages / tile_order=0, five fresh processes, 30 warmups and 200
CUDA Graph samples per process. Both TLE variants used the same FlagGems typed
pipe kernel; the candidate changed only the pipe position from absolute
iteration to the new cursor API. The table reports the median of per-process
average latency.

Scenario Absolute iteration Cursor Change
CUDA Graph latency 0.0373608 ms 0.0372621 ms 0.99736x (0.26% lower)
Dynamic instructions 2,639,002 2,424,162 -214,840 (-8.14%)

NCU 2025.4.1 was run under authorized container root because the H20 driver
requires profiling privilege. Correctness, normal tests, and independent
timing ran as the mapped non-root user. The two profiled outputs were bitwise
equal.

NCU launch/resource metric Absolute iteration Cursor
Grid / block 78 / 256 78 / 256
Registers per thread 255 255
Dynamic shared memory 141,444 B 141,444 B
Register spills 0 0
Waves per SM 1 1
TMA pipe activity 0.62% 0.62%

The NCU replay duration is diagnostic only and is not used as the latency
claim. The 0.26% latency result comes from the separate five-process CUDA
Graph gate.

Five-process CUDA Graph averages — absolute values used for the median
Replica Absolute iteration (ms) Cursor (ms)
0 0.03731088 0.03711344
1 0.03723168 0.03785536
2 0.03749408 0.03694288
3 0.04047392 0.03726208
4 0.03736080 0.04052672

Files changed

  • python/triton/experimental/tle/language/gpu/types.py
    — cursor type/value, initialization, advance, and endpoint consumption.
  • python/triton/experimental/tle/language/pipe.py
    — public cursor alias.
  • python/triton/experimental/tle/language/__init__.py
    — top-level export.
  • python/test/tle/unit/test_tle.py
    — frontend/type/capacity tests.
  • python/test/tle/integration/test_tle_pipe_cursor.py
    — TTIR loop-state and arithmetic regression.

Known limits

  • This is an opt-in language/API change, not an automatic compiler induction
    rewrite. Existing kernels keep their current code generation until migrated.
  • Performance and NCU evidence are bounded to one companion FP8 kernel,
    shape, and configuration. The dynamic-instruction result is strong mechanism
    evidence; the observed 0.26% latency result must not be generalized to all
    TLE kernels.
  • Multi-reader, one-shot, close, and distributed pipe performance were not
    benchmarked. Existing absolute-iteration behavior remains the compatibility
    path for those users.

@CLAassistant

CLAassistant commented Aug 14, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants