[Bug Report] HSACOError & FileNotFoundError when compiling Triton kernel on Hygon BW1000 HCU

[xiaohao668]

Environment Information

• Image: flagtree-hcu3.6-py310-torch2.4.1-ubuntu22.04:202604-base
• Hardware: Hygon BW1000 (arch: gfx936)
• GPU Count: 8
• Installation Method: Source-free installation following documents/install_hcu_cn.md (Section 2.1)
• Component Versions:
  • flagtree: 0.6.0+hcu3.6
  • triton: 3.6.0
  • TRITON_HIP_CLANG_PATH: /opt/dtk/dcc/bin/clang (clang-17.0.0 from DTK driverruntime branch)
    Problem Description

Reproduction Steps

Launch container

docker run ... flagtree-hcu3.6-py310-torch2.4.1-ubuntu22.04:202604-base

Inside container (pip install can be skipped as the image is pre-installed)

pip install flagtree===0.6.0+hcu3.6
python3 python/tutorials/01-vector-add.py
Error Log (Key Part)
File ".../triton/backends/hcu/compiler_hcu.py", line 756, in make_amdgcn
with open(asm_file, "r") as fd_out:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpXXXX.amdgcn'
...
HSACOError: HSACO error: File operation failed: ...

Clang stderr output (unknown arguments):
clang (LLVM option parsing): Unknown command line argument '-check-valu-data-forward-hazards=0'
clang (LLVM option parsing): Unknown command line argument '-disable-cluster-lds-memops=true'

Root Cause Analysis

1. The make_amdgcn function of HCU backend passes several HCU-proprietary LLVM flags via _get_clang_args when compiling LLVM IR (.ll) to amdgcn assembly (.amdgcn). These flags are only supported by Hygon-customized LLVM, not standard upstream clang.
2. The path_to_rocm_clang() resolves to /opt/dtk/dcc/bin/clang (standard clang-17), which cannot recognize the HCU-specific flags, leading to compilation failure and missing .amdgcn output file. This triggers FileNotFoundError and is finally wrapped into HSACOError.
3. The default expected clang path for path_to_rocm_clang() is /opt/dtk/aillvm/bin/clang-18 (aillvm branch), which does not exist in the current image.
4. The built-in hcu-llvm22 (~/.flagtree/hcu/hcu-llvm22-xxx) only provides backend tools (llc/opt/lld/mlir-opt) without clang frontend. Itsllc also lacks support for the above HCU-specific flags and is only used for build-time processing, not runtime kernel compilation.

Key Verification Evidence

1. Standard clang-17 has no HCU proprietary flag support:
   strings /opt/dtk/dcc/bin/clang-17 | grep -c "check-valu-data-forward-hazards"

Output: 0

2. Built-in hcu-llvm22 llc lacks all related HCU flags:
   llc --help-hidden | grep -E "check-valu-data-forward-hazards|disable-cluster-lds-memops|hcu-pre-emit-load-store-opt|support-768-vgprs"

No output

3. Expected aillvm clang directory missing:
   ls /opt/dtk/aillvm/bin/clang-18

ls: cannot access '/opt/dtk/aillvm/bin/clang-18': No such file or directory

Questions for Maintainers

• Is the 202604-base image missing the Hygon-customized clang (aillvm clang-18) required for runtime Triton kernel compilation?
• What is the correct compiler toolchain configuration for running Triton HCU kernels on the current official image?
• Is there a temporary workaround to bypass the unsupported hardcoded HCU compilation flags for standard clang?
• Will the official base image be updated to include the complete customized LLVM/clang toolchain for HCU backend?

[zhzhcookie]

Please try flagtree===0.6.1a1+hcu3.6.

[xiaohao668]

Please try flagtree===0.6.1a1+hcu3.6.

Thanks for the quick reply! I installed flagtree===0.6.1a1+hcu3.6, but unfortunately the issue still reproduces with the same HSACOError when running
python/tutorials/01-vector-add.py.

After digging in, the root cause looks like the :202604-base image doesn't ship an HCU-aware clang, so the backend falls back to the stock DTK clang-17, which doesn't recognize
the HCU-specific -mllvm options.

Evidence

$ python3 -m pip show flagtree | grep Version
Version: 0.6.1a1+hcu3.6

$ python3 -c "from triton.backends.hcu.compiler_hcu import HIPBackend; print(HIPBackend.path_to_rocm_clang())"
/opt/dtk/dcc/bin/clang

$ /opt/dtk/dcc/bin/clang --version
clang version 17.0.0 (http://10.4.0.1/dcutoolkit/driverruntime/llvm-project.git ...)

Stock clang-17 has zero HCU backend options compiled in:

$ strings /opt/dtk/dcc/bin/clang-17 | grep -c "check-valu-data-forward-hazards"
0

The default lookup also points to a clang that isn't present in the image:

$ env -u TRITON_HIP_CLANG_PATH python3 -c "from triton.backends.hcu.compiler_hcu import HIPBackend; print(HIPBackend.path_to_rocm_clang())"
Exception: ROCm compiler not found under /opt/dtk/aillvm/bin/clang. Set 'TRITON_HIP_CLANG_PATH' to its path.

→ /opt/dtk/aillvm does not exist in the :202604-base image

I also confirmed the new wheel does not bundle a clang binary.

What I've already tried

To rule out a configuration issue, I systematically patched compiler_hcu.py and tested several workarounds — all failed due to the same underlying version mismatch between
libtriton (linked against LLVM 22) and the stock clang-17 (LLVM 17):

• Removing the HCU -mllvm options → fails at make_hsaco (undefined symbol: llvm.amdgcn.raw.ptr.buffer.load.v4f32 — LLVM-22 intrinsics not lowered by the LLVM-17 backend).
• Routing make_amdgcn through the hcu-llvm22 llc → intrinsics get lowered correctly, but make_hsaco's clang-17 assembler rejects the LLVM-22 .amdhsa_* directives.

Question

It seems the :202604-base image simply doesn't include the HCU-aware clang. Where should the HCU-customized clang (LLVM 22 + HCU backend) be obtained from? Is there a missing
setup step (e.g., an additional package, or a bind-mount into /opt/dtk/aillvm) that the install docs don't mention?

Environment: flagtree-hcu3.6-py310-torch2.4.1-ubuntu22.04:202604-base, BW1000 (gfx936).

Thanks!