Because there is a fixed set of inputs that are reused by flashinfer-bench, a kernel can just remember what it previously output for each input and return that without needing to do any computation, resulting in much lower measured latency than is realistic.

I made a minimal reproducible example based on flashinfer-bench-starter-kit: https://github.com/tripplyons/flashinfer-bench-bug