Some recent cuda commit broke the enzyme.jl tests?

[wsmoses]

Not sure what since CI has been red (and seemingly ignored)?

@maleadt @vchuravy

Test           (Worker) │ time (s) │ GC (s) │ Alloc (MB) │ RSS (MB) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
--
  | extensions/enzyme   (1) │   851.38 │   0.01 │       0.00 │   116.00 │  18.68 │  2.2 │  106925.54 │  4345.29 │
  |  
  | Test Summary:                   \| Pass  Error  Total      Time
  | Overall                       \|   17      3     20  14m41.5s
  | extensions/enzyme           \|   17      3     20  14m11.4s
  | compiler_job_from_backend \|    1             1      0.8s
  | Make_zero                 \|    2             2     24.8s
  | Forward Kernel            \|           1      1   1m40.3s
  | Reverse Kernel            \|    3             3   6m39.7s
  | Forward Fill!             \|    2             2     21.7s
  | Reverse Fill!             \|    3             3     40.5s
  | Forward allocate          \|           1      1      1.9s
  | Reverse allocate          \|           1      1      3.8s
  | Forward broadcast         \|                  0      0.2s
  | Forward sum               \|    1             1     36.2s
  | Reverse sum               \|    1             1     28.8s
  | Forward setadd            \|    1             1   1m00.4s
  | setadd sum                \|    2             2     44.0s
  | Reverse sum abs2          \|    1             1     44.6s
  | FAILURE
  |  
  | Error in testset Forward Kernel:
  | Error During Test at /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/extensions/enzyme.jl:32
  | Got exception outside of a @test
  | MethodError: no method matching forward(::EnzymeCore.EnzymeRules.FwdConfigWidth{1, true, false, false, false}, ::EnzymeCore.Const{typeof(cufunction)}, ::Type{EnzymeCore.Const{CUDACore.HostKernel{typeof(Main.var"##extensions/enzyme#241".square_kernel!), Tuple{CuDeviceVector{Float32, 1}}}}}, ::EnzymeCore.Const{typeof(Main.var"##extensions/enzyme#241".square_kernel!)}, ::EnzymeCore.Const{Type{Tuple{CuDeviceVector{Float32, 1}}}})
  | The function `forward` exists, but no method is defined for this combination of argument types.
  |  
  | Closest candidates are:
  | forward(::EnzymeCore.EnzymeRules.FwdConfig, ::EnzymeCore.Const{typeof(hypot)}, ::Any, ::EnzymeCore.Annotation, ::EnzymeCore.Annotation, ::EnzymeCore.Annotation, ::EnzymeCore.Annotation...) where N
  | @ Enzyme ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/internal_rules/math.jl:6
  | forward(::Any, ::EnzymeCore.Const{typeof(finalizer)}, ::Any, ::EnzymeCore.Const, ::Any)
  | @ Enzyme ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/internal_rules/core.jl:408
  | forward(::Any, ::EnzymeCore.Const{typeof(cufunction)}, ::Type{<:EnzymeCore.Duplicated}, ::EnzymeCore.Const{F}, ::EnzymeCore.Const{TT}; kwargs...) where {F, TT}
  | @ EnzymeCoreExt /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/CUDACore/ext/EnzymeCoreExt.jl:39
  | ...
  |  
  | Stacktrace:
  | [1] custom_rule_method_error
  | @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/rules/customrules.jl:976 [inlined]
  | [2] kernel_compile
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/CUDACore/src/compiler/execution.jl:60 [inlined]
  | [3] macro expansion
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/CUDACore/src/compiler/execution.jl:183 [inlined]
  | [4] square!
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/extensions/enzyme.jl:28 [inlined]
  | [5] fwddiffejulia_square__58054wrap
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/extensions/enzyme.jl:0
  | [6] macro expansion
  | @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:6856 [inlined]
  | [7] enzyme_call
  | @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:6326 [inlined]
  | [8] ForwardModeThunk
  | @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:6226 [inlined]
  | [9] autodiff
  | @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/Enzyme.jl:688 [inlined]
  | [10] autodiff
  | @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/Enzyme.jl:577 [inlined]
  | [11] autodiff(mode::EnzymeCore.ForwardMode{false, EnzymeCore.FFIABI, false, false, false}, f::typeof(Main.var"##extensions/enzyme#241".square!), args::EnzymeCore.Duplicated{CuArray{Float32, 1, CUDACore.DeviceMemory}})
  | @ Enzyme ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/Enzyme.jl:549
  | [12] macro expansion
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/extensions/enzyme.jl:37 [inlined]
  | [13] macro expansion
  | @ ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.11/julia-1.11-latest-linux-x86_64/share/julia/stdlib/v1.11/Test/src/Test.jl:1709 [inlined]
  | [14] top-level scope
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/extensions/enzyme.jl:33
  | [15] include(mod::Module, _path::String)
  | @ Base ./Base.jl:562
  | [16] include
  | @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/packages/ParallelTestRunner/kZFbj/src/ParallelTestRunner.jl:420 [inlined]
  | [17] macro expansion
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/setup.jl:105 [inlined]
  | [18] macro expansion
  | @ ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.11/julia-1.11-latest-linux-x86_64/share/julia/stdlib/v1.11/Test/src/Test.jl:1709 [inlined]
  | [19] macro expansion
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/setup.jl:105 [inlined]
  | [20] macro expansion
  | @ ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.11/julia-1.11-latest-linux-x86_64/share/julia/stdlib/v1.11/Test/src/Test.jl:1709 [inlined]
  | [21] macro expansion
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/setup.jl:104 [inlined]
  | [22] macro expansion
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/CUDACore/src/utilities.jl:35 [inlined]
  | [23] macro expansion
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/CUDACore/src/memory.jl:906 [inlined]
  | [24] top-level scope
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/setup.jl:103
  | Error in testset Forward allocate:
  | Error During Test at /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/extensions/enzyme.jl:89
  | Got exception outside of a @test
  | UndefVarError: `Mem` not defined in `CUDA`
  | Suggestion: check for spelling errors or missing imports.
  | Stacktrace:
  | [1] alloc
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/extensions/enzyme.jl:87 [inlined]
  | [2] fwddiffejulia_alloc_202532wrap
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/extensions/enzyme.jl:0
  | [3] macro expansion
  | @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:6856 [inlined]
  | [4] enzyme_call
  | @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:6326 [inlined]
  | [5] ForwardModeThunk
  | @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:6226 [inlined]
  | [6] autodiff
  | @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/Enzyme.jl:688 [inlined]
  | [7] autodiff(mode::EnzymeCore.ForwardMode{true, EnzymeCore.FFIABI, false, false, false}, f::typeof(Main.var"##extensions/enzyme#241".alloc), ::Type{EnzymeCore.Duplicated}, args::EnzymeCore.Const{Int64})
  | @ Enzyme ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/Enzyme.jl:557
  | [8] macro expansion
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/extensions/enzyme.jl:90 [inlined]
  | [9] macro expansion
  | @ ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.11/julia-1.11-latest-linux-x86_64/share/julia/stdlib/v1.11/Test/src/Test.jl:1709 [inlined]
  | [10] top-level scope
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/extensions/enzyme.jl:90
  | [11] include(mod::Module, _path::String)
  | @ Base ./Base.jl:562
  | [12] include
  | @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/packages/ParallelTestRunner/kZFbj/src/ParallelTestRunner.jl:420 [inlined]
  | [13] macro expansion
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/setup.jl:105 [inlined]
  | [14] macro expansion
  | @ ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.11/julia-1.11-latest-linux-x86_64/share/julia/stdlib/v1.11/Test/src/Test.jl:1709 [inlined]
  | [15] macro expansion
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/setup.jl:105 [inlined]
  | [16] macro expansion
  | @ ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.11/julia-1.11-latest-linux-x86_64/share/julia/stdlib/v1.11/Test/src/Test.jl:1709 [inlined]
  | [17] macro expansion
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/setup.jl:104 [inlined]
  | [18] macro expansion
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/CUDACore/src/utilities.jl:35 [inlined]
  | [19] macro expansion
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/CUDACore/src/memory.jl:906 [inlined]
  | [20] top-level scope
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/setup.jl:103
  | Error in testset Reverse allocate:
  | Error During Test at /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/extensions/enzyme.jl:97
  | Got exception outside of a @test
  | UndefVarError: `Mem` not defined in `CUDA`
  | Suggestion: check for spelling errors or missing imports.
  | Stacktrace:
  | [1] alloc
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/extensions/enzyme.jl:87 [inlined]
  | [2] augmented_julia_alloc_204053wrap
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/extensions/enzyme.jl:0
  | [3] macro expansion
  | @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:6856 [inlined]
  | [4] enzyme_call
  | @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:6326 [inlined]
  | [5] (::Enzyme.Compiler.AugmentedForwardThunk{Ptr{Nothing}, EnzymeCore.Const{typeof(Main.var"##extensions/enzyme#241".alloc)}, EnzymeCore.Duplicated{CuArray{Float32, 1}}, Tuple{EnzymeCore.Const{Int64}}, 1, true, @NamedTuple{1::Core.LLVMPtr{Tuple{}, 0}, 2::Core.LLVMPtr{Tuple{}, 0}, 3::Core.LLVMPtr{Tuple{}, 0}, 4, 5, 6, 7, 8, 9, 10::Core.LLVMPtr{Tuple{}, 0}, 11}})(fn::EnzymeCore.Const{typeof(Main.var"##extensions/enzyme#241".alloc)}, args::EnzymeCore.Const{Int64})
  | @ Enzyme.Compiler ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:6274
  | [6] macro expansion
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/extensions/enzyme.jl:99 [inlined]
  | [7] macro expansion
  | @ ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.11/julia-1.11-latest-linux-x86_64/share/julia/stdlib/v1.11/Test/src/Test.jl:1709 [inlined]
  | [8] top-level scope
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/extensions/enzyme.jl:98
  | [9] include(mod::Module, _path::String)
  | @ Base ./Base.jl:562
  | [10] include
  | @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/packages/ParallelTestRunner/kZFbj/src/ParallelTestRunner.jl:420 [inlined]
  | [11] macro expansion
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/setup.jl:105 [inlined]
  | [12] macro expansion
  | @ ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.11/julia-1.11-latest-linux-x86_64/share/julia/stdlib/v1.11/Test/src/Test.jl:1709 [inlined]
  | [13] macro expansion
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/setup.jl:105 [inlined]
  | [14] macro expansion
  | @ ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.11/julia-1.11-latest-linux-x86_64/share/julia/stdlib/v1.11/Test/src/Test.jl:1709 [inlined]
  | [15] macro expansion
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/setup.jl:104 [inlined]
  | [16] macro expansion
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/CUDACore/src/utilities.jl:35 [inlined]
  | [17] macro expansion
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/CUDACore/src/memory.jl:906 [inlined]
  | [18] top-level scope
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/setup.jl:103
  | ERROR: LoadError: Test run finished with errors
  | in expression starting at /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/runtests.jl:223
  | ERROR: Package CUDA errored during testing
  | Stacktrace:
  | [1] pkgerror(msg::String)
  | @ Pkg.Types ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.11/julia-1.11-latest-linux-x86_64/share/julia/stdlib/v1.11/Pkg/src/Types.jl:68
  | [2] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool)
  | @ Pkg.Operations ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.11/julia-1.11-latest-linux-x86_64/share/julia/stdlib/v1.11/Pkg/src/Operations.jl:2128
  | [3] test
  | @ ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.11/julia-1.11-latest-linux-x86_64/share/julia/stdlib/v1.11/Pkg/src/Operations.jl:2011 [inlined]
  | [4] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool, kwargs::@Kwargs{io::IOContext{IO}})
  | @ Pkg.API ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.11/julia-1.11-latest-linux-x86_64/share/julia/stdlib/v1.11/Pkg/src/API.jl:481
  | [5] test(pkgs::Vector{Pkg.Types.PackageSpec}; io::IOContext{IO}, kwargs::@Kwargs{coverage::Bool, test_args::Cmd})
  | @ Pkg.API ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.11/julia-1.11-latest-linux-x86_64/share/julia/stdlib/v1.11/Pkg/src/API.jl:159
  | [6] test(; name::Nothing, uuid::Nothing, version::Nothing, url::Nothing, rev::Nothing, path::Nothing, mode::Pkg.Types.PackageMode, subdir::Nothing, kwargs::@Kwargs{coverage::Bool, test_args::Cmd})
  | @ Pkg.API ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.11/julia-1.11-latest-linux-x86_64/share/julia/stdlib/v1.11/Pkg/src/API.jl:174
  | [7] top-level scope
  | @ none:27
  | 🚨 Error: The command exited with status 1

[maleadt]

Please don't simply dump a stack trace on an issue tracker, without any explanation. You haven't included a MWE, and nothing here points at CUDA.jl. This is not actionable to any CUDA.jl maintainer, and can only be fixed by Enzyme.jl developers. If anything, it only helps to convince me this code should move into Enzyme.jl as a CUDA.jl extension instead.

and seemingly ignored

Yes, it kept failing so I disabled it, and tagged you in the message: 209f97e

[kshyatt]

Since I keep bothering both CUDA.jl and Enzyme.jl why don't I take a look at this one

[wsmoses]

sorry didn't notice that, and more than a year old [and that's not the current error we see]. In the future when disabled, can we open an error message here for persistence.

The current stacktrace above was fairly instructive to me, and seems to definitely point to a change in cuda.jl, and actionable by a cuda maintainer. See

 | UndefVarError: `Mem` not defined in `CUDA`
  | Suggestion: check for spelling errors or missing imports.
  | Stacktrace:
  | [1] alloc
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/extensions/enzyme.jl:87 [inlined]
  | [2] augmented_julia_alloc_204053wrap
  | @ /var/lib/buildkite-agent/builds/gpuci-1/julialang/cuda-dot-jl/test/extensions/enzyme.jl:0

that just seems that CUDA changed the mem definition for a helper function?

[maleadt]

Your test is bad:

CUDA.jl/test/extensions/enzyme.jl

Line 87 in 642cf8d

alloc(x) = CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}(undef, (x,))

The Mem submodule was removed back in 2024: #2335

[wsmoses]

sure, but the test was always like that as far as I can tell? So presumably whatever removed it didn't also update the test?

also regardless seemingly relevant to this repo and cuda developers :)

[maleadt]

Mem was deprecated for 2 years, and when we release 6.0 the deprecation turned in to a removal.

[wsmoses]

yeah, so that's probably what broke it then, what's the replacement?

The other error from the log seems to be that the forward mode custom derivative for cufunction defined in the extension is no longer hit properly?

Anything about the api or usage which would've changed there (we can fix the mem first, then figure out the cufunction one second). @kshyatt you still should have the lock on this, not trying to remove your assignment lol