Merge remote-tracking branch 'origin/main' into copilot/add-custom-logging-callback

Author: tianleiwu

.agents/skills/cuda-attention-kernel-patterns/SKILL.md

@@ -0,0 +1,111 @@
+---
+name: cuda-cutlass-fmha-incremental-rebuild
+description: >
+  Use when rebuilding ONNX Runtime CUDA after editing CUTLASS fused-MHA headers
+  (onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/*.h such as kernel_forward.h or
+  fmha_launch_template.h), or when a header edit "passed" an incremental build but
+  test behavior did not change. Explains the nvcc depfile gotcha that produces stale
+  Memory-Efficient-Attention (MEA) kernels and binaries, and how to force a correct
+  recompile. Also covers disk-space frugality on shared GPU dev boxes.
+---
+
+# Incremental rebuilds silently use STALE CUTLASS fused-MHA kernels
+
+> The **general** false-green principles (stale binary, wrong-artifact mtime) are summarised
+> in the `ort-test` skill's "False-green taxonomy". This skill is the CUDA/CUTLASS-specific
+> detail.
+
+## The gotcha (verification-integrity bug)
+
+`nvcc`-generated depfiles do **not** track the CUTLASS fused-MHA headers under
+`onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/` (e.g. `kernel_forward.h`,
+`fmha_launch_template.h`). These headers are `#include`d by the `fmha_sm*.cu`
+translation units, but the build system does not record that dependency.
+
+Consequence: after you edit one of those headers, an **incremental** `build.sh`:
+
+- does **not** recompile `fmha_sm*.cu`,
+- reports `[100%] Built target ...` and exits 0,
+- leaves the recompiled artifacts — the `fmha_sm*.cu.o` objects and the
+  `libonnxruntime_providers_cuda.so` they link into — **unchanged** (same `mtime` as
+  the pre-edit build).
+
+(Do **not** use the gtest test-exe mtime as the stale symptom: in the shared-provider
+build the exe `dlopen`s the `.so` and is **not** relinked, so its mtime stays old even
+after a *correct* rebuild — see "How to confirm" below. The reliable diagnostic signal
+is the `fmha_sm*.cu.o` / `.so` mtime.)
+
+So your "successful" rebuild is running the **old** kernel. Tests that should now
+pass (or fail) reflect the previous code, not your edit. This silently invalidates
+any FAIL→PASS / PASS→FAIL verification.
+
+## The fix — force recompile the .cu units
+
+Before rebuilding after editing any `cutlass_fmha/*.h` header:
+
+```bash
+touch onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/*.cu
+```
+
+Then run the normal build command. This forces the `fmha_sm*.cu` translation units
+(and downstream binaries) to recompile against your header change.
+
+## How to confirm the rebuild was real (don't trust "[100%] Built")
+
+Confirm that the artifact which actually **links** the recompiled `fmha_sm*.cu.o`
+is newer than your header edit.
+
+⚠️ **Do NOT just check the test EXE mtime — it can falsely flag a good build as
+stale.** In the shared-provider build configuration (the default here), the CUDA
+execution provider is a **shared module**: the recompiled `fmha_sm*.cu.o` link into
+`libonnxruntime_providers_cuda.so`, and the `onnxruntime_provider_test` executable
+**dlopens** that `.so` — it is **not relinked**. So after a *correct* rebuild the
+test exe `mtime` stays **old** while the `.so` advances. Checking the exe alone
+would wrongly conclude the build was stale.
+
+Check the right artifact for your link mode:
+
+- **Shared-provider build (default):** the `.so` that links the recompiled `.o` —
+  `build/<dir>/<cfg>/libonnxruntime_providers_cuda.so`
+- **Statically-linked provider:** the test exe itself (`onnxruntime_provider_test`)
+
+Safest check — `stat` both the recompiled object and the `.so`, and confirm BOTH
+are newer than the header edit:
+
+```bash
+stat -c '%y %n' onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/kernel_forward.h
+# in your build dir, e.g. build/Debug_quickbuild/Debug/:
+stat -c '%y %n' libonnxruntime_providers_cuda.so
+# and the actual recompiled object (path varies by build dir):
+find . -name 'fmha_sm80.cu.o' -exec stat -c '%y %n' {} +
+```
+
+If the `.so` (and the `fmha_sm*.cu.o`) timestamps are older than (or equal to) the
+header edit, the build was stale — `touch` the `.cu` files and rebuild. The most
+reliable signal of all is behavioral: a test that was failing now passes (a stale
+binary cannot flip its result).
+
+## Related: pick the right test binary
+
+This is the **CUDA/CUTLASS instance of false-green mode 1** (zero-match / wrong binary) —
+see the `ort-test` skill's "False-green taxonomy" for the general principle. In short:
+attention/MEA/Flash boundary gtests (e.g. `FlashStructuralEmptyRows*`,
+`Attention_Causal_NonPadKVSeqLen_MEA_*`) live in **`onnxruntime_provider_test`**, which CI
+runs; `onnxruntime_test_all` does not contain them and gives a false green. Verify the
+MEA/Flash boundary fix against `onnxruntime_provider_test`.
+
+## Related: disk frugality on shared GPU dev boxes
+
+Full ORT CUDA builds are large (test binaries ~1 GB each; a build dir can reach
+tens of GB). On a shared box, `/home` filling to 100% makes builds fail in
+non-obvious places — e.g. `git submodule sync` reporting `No space left on device`
+or a `config.lock` error, not an obvious "disk full" at the compile step.
+
+Before a big rebuild, check free space and clean only clearly-stale, regenerable
+build directories (old dated experiment dirs). Never delete another agent's active
+build dir or anything ambiguous:
+
+```bash
+df -h /home
+du -sh build/* | sort -h
+```