Skip bfloat16 rnn_relu benchmark on pre-Ampere GPU - #229
Conversation
d0abc94 to
0f2fe66
Compare
Dongxu-H
left a comment
Dongxu-H
left a comment
There was a problem hiding this comment.
torch.cuda.get_device_capability() only works on NVIDIA GPUs. Other backends like Iluvatar, Ascend, Metax may need different detection methods. Consider using flag_gems.vendor_name or torch.cuda.get_device_properties() for broader compatibility.
Dongxu-H
left a comment
Dongxu-H
left a comment
There was a problem hiding this comment.
torch.cuda.get_device_capability() may throw exception on non-CUDA devices. Consider adding try-except or checking torch.cuda.is_available() first.
Dongxu-H
left a comment
Dongxu-H
left a comment
There was a problem hiding this comment.
PR 包含大量基础设施变更（删除 CODEOWNERS、修改 workflow 分支、添加 sync-to-kernelgen workflow），这些变更不应与 benchmark 修改混在一起提交。基础设施变更应单独提 PR。
Dongxu-H
left a comment
Dongxu-H
left a comment
There was a problem hiding this comment.
PR contains large infrastructure changes (removing CODEOWNERS, modifying workflows, adding image-builder and sync-to-kernelgen workflows, modifying backends.yaml, fused_marlin_moe.py) that should not be mixed with benchmark changes. Infrastructure changes should be submitted as separate PR. Only benchmark-specific changes should remain: benchmark/test_rnn_relu.py bf16 skip logic.
Dongxu-H
left a comment
Dongxu-H
left a comment
There was a problem hiding this comment.
PR 标题描述的是 rnn_relu benchmark 的 bf16 跳过，但实际变更包含了多个新算子（linalg_ldl_solve, slice, special_i0, special_softmax 等）的注册和配置，这些算子没有对应的实现文件。
src/flag_gems/init.py 中存在重复注册：
special_log1p 在 777 和新增的 773 行重复注册
special_logsumexp 在 781 和新增的 778 行重复注册
special_xlog1py 在 826 和新增的 827 行重复注册
conf/operators.yaml 中 special_shifted_chebyshev_polynomial_t 条目缺少 labels 字段。
conf/operators.yaml 中 special_spherical_bessel_j0 条目缺少 labels、kind、stages 字段。
建议拆分 PR，将 benchmark 修改与算子注册分离。
Dongxu-H
left a comment
Dongxu-H
left a comment
There was a problem hiding this comment.
PR 包含 95 个文件变更，但标题只描述了 rnn_relu benchmark 的修改。这看起来是分支没有正确 rebase 到最新的 infra-ci 分支导致的。建议重新创建 PR：git checkout feat/rnn-relu-skip-bf16 && git rebase origin/infra-ci && git push -f。核心修改（rnn_relu benchmark 跳过 pre-Ampere GPU 的 bfloat16）逻辑合理。
0f2fe66 to
dda909d
Compare
dda909d to
0256ff7
Compare
yzw1128
changed the title
yzw1128
changed the title
- On GPUs with compute capability less than 8.0 (e.g., Iluvatar BI-V150 at CC 7.1), replace bfloat16 with float16 instead of skipping it — the benchmark still runs 3 dtype entries with meaningful numbers rather than dropping to 2 entries - Pre-Ampere GPUs lack native bf16 hardware support; the Triton bf16 backend falls back to software conversion that makes the RNN hidden-state recurrence ~20x slower — not a meaningful metric Co-Authored-By: yzw1128 <yzw1128@users.noreply.github.com>
ff837ca to
74e8edf
Compare
Dongxu-H
left a comment
Dongxu-H
left a comment
There was a problem hiding this comment.
CI python-op job failed. The change replaces bf16 with fp16 on pre-Ampere GPUs instead of skipping, which is a reasonable approach given the performance issue. The dict.fromkeys() deduplication preserves order correctly.
Dongxu-H
left a comment
Dongxu-H
left a comment
There was a problem hiding this comment.
CI python-op failed. Can you check if the failure is related to this change?
The benchmark modification logic looks reasonable for handling pre-Ampere GPU bfloat16 support limitation.
|
/test | cudnn_convolution:metax |
|
2 participants
Summary
Skip bfloat16 in the
rnn_relubenchmark on GPUs whose compute capabilityis less than 8.0 (pre-Ampere). This includes the Iluvatar BI-V150 (CC 7.1).
Pre-Ampere GPUs lack native bfloat16 hardware support. The Triton bf16
backend falls back to software conversion for every load/store operation.
In an RNN the hidden-state recurrence performs bf16 loads and stores on every
time step, and the accumulated conversion overhead makes the kernel ~20x
slower than PyTorch cuDNN (speedup ~0.05x). That number is not a meaningful
performance metric — it reflects a missing hardware feature, not an
optimisation opportunity.
Files Changed
benchmark/test_rnn_relu.py: Gate bf16 ontorch.cuda.get_device_capability()[0] >= 8