Skip to content

Fix PSO benchmark : skip unstable BBOB functions - #1613

Open
AdityaPandeyCN wants to merge 2 commits into
SciML:masterfrom
AdityaPandeyCN:pso_patch
Open

Fix PSO benchmark : skip unstable BBOB functions#1613
AdityaPandeyCN wants to merge 2 commits into
SciML:masterfrom
AdityaPandeyCN:pso_patch

Conversation

@AdityaPandeyCN

@AdityaPandeyCN AdityaPandeyCN commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Pin CUDA runtime via LocalPreferences extras and exclude f4, f7, f10 which cause segfaults or illegal memory access on GPU.

Checklist

  • Appropriate tests were added
  • Any code changes were done in a way that does not break public API
  • All documentation related to code changes were updated
  • The new code follows the
    contributor guidelines, in particular the SciML Style Guide and
    COLPRAC.
  • Any new documentation only uses public API

Additional context

The PR also adds updated Manifest.toml

Pin CUDA runtime via LocalPreferences extras and exclude f4, f7, f10
which cause segfaults or illegal memory access on GPU.

Signed-off-by: AdityaPandeyCN <adityapand3y666@gmail.com>
@ChrisRackauckas

Copy link
Copy Markdown
Member

What's the cudaruntime thing? That's probably not the way to solve it.

Also, why was the benchmark okay with these before but not now?

@AdityaPandeyCN

Copy link
Copy Markdown
Contributor Author

What's the cudaruntime thing? That's probably not the way to solve it.

I aimed to resolve this failure from earlier benchmark run

ERROR: Compute capability 7.0.0 is not supported by CUDA 13.2.0
Stacktrace:
  [1] (::Weave.var"#32#34"{String, Module, String, Bool, Task, Base.PipeEndpoint, Base.PipeEndpoint, Base.PipeEndpoint})()
    @ Weave ~/.julia/packages/Weave/f7Ly3/src/run.jl:222
  [2] task_local_storage(body::Weave.var"#32#34"{String, Module, String, Bool, Task, Base.PipeEndpoint, Base.PipeEndpoint, Base.PipeEndpoint}, key::Symbol, val::String)
    @ Base ./task.jl:304
  [3] capture_output(code::String, mod::Module, path::String, error::Bool, report::Weave.Report)
    @ Weave ~/.julia/packages/Weave/f7Ly3/src/run.jl:216
  [4] #29
    @ ~/.julia/packages/Weave/f7Ly3/src/run.jl:191 [inlined]
  [5] _broadcast_getindex_evalf
    @ ./broadcast.jl:709 [inlined]
  [6] _broadcast_getindex
    @ ./broadcast.jl:682 [inlined]
  [7] getindex
    @ ./broadcast.jl:636 [inlined]
  [8] macro expansion
    @ ./broadcast.jl:1004 [inlined]
  [9] macro expansion
    @ ./simdloop.jl:77 [inlined]
 [10] copyto!
    @ ./broadcast.jl:1003 [inlined]
 [11] copyto!
    @ ./broadcast.jl:956 [inlined]
 [12] copy
    @ ./broadcast.jl:928 [inlined]
 [13] materialize(bc::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Nothing, Weave.var"#29#30"{Weave.Report, Module, Bool, String}, Tuple{Vector{String}}})
    @ Base.Broadcast ./broadcast.jl:903
 [14] run_code(doc::Weave.WeaveDoc, chunk::Weave.CodeChunk, report::Weave.Report, mod::Module)
    @ Weave ~/.julia/packages/Weave/f7Ly3/src/run.jl:192
 [15] eval_chunk(doc::Weave.WeaveDoc, chunk::Weave.CodeChunk, report::Weave.Report, mod::Module)
    @ Weave ~/.julia/packages/Weave/f7Ly3/src/run.jl:263
 [16] run_chunk(chunk::Weave.CodeChunk, doc::Weave.WeaveDoc, report::Weave.Report, mod::Module)
    @ Weave ~/.julia/packages/Weave/f7Ly3/src/run.jl:133
 [17] run_doc(doc::Weave.WeaveDoc; doctype::String, out_path::String, args::Dict{Symbol, String}, mod::Nothing, fig_path::Nothing, fig_ext::Nothing, cache_path::String, cache::Symbol)
    @ Weave ~/.julia/packages/Weave/f7Ly3/src/run.jl:74
 [18] weave(source::String; doctype::String, informat::Nothing, out_path::String, args::Dict{Symbol, String}, mod::Nothing, fig_path::Nothing, fig_ext::Nothing, cache_path::String, cache::Symbol, template::Nothing, css::Nothing, highlight_theme::Nothing, pandoc_options::Vector{String}, latex_cmd::Vector{String}, keep_unicode::Bool)
    @ Weave ~/.julia/packages/Weave/f7Ly3/src/Weave.jl:176
 [19] weave
    @ ~/.julia/packages/Weave/f7Ly3/src/Weave.jl:124 [inlined]
 [20] weave_file(folder::String, file::String, build_list::Tuple{Symbol, Symbol})
    @ SciMLBenchmarks ~/_work/SciMLBenchmarks.jl/SciMLBenchmarks.jl/src/SciMLBenchmarks.jl:61
 [21] top-level scope
    @ none:5

caused by: LoadError: Compute capability 7.0.0 is not supported by CUDA 13.2.0

Also, why was the benchmark okay with these before but not now?

It ran for me on T4 colab GPU but for utkarsh's cluster run of this benchmark these functions were problem, precisely this.
Affected (3/19) F4 Buche-Rastrigin, F10 Ellipsoidal-2 (catchable illegal-access), F7 Step-Ellipsoidal (hard segfault)

Signed-off-by: AdityaPandeyCN <adityapand3y666@gmail.com>
@AdityaPandeyCN

Copy link
Copy Markdown
Contributor Author

@ChrisRackauckas Benchmarks are finally reproduced by the CI :)

@AdityaPandeyCN

Copy link
Copy Markdown
Contributor Author

@ChrisRackauckas Can you take a look at the artifacts?
https://github.com/SciML/SciMLBenchmarks.jl/actions/runs/29597613342/artifacts/8435117115

I probably need more tuning but let me know your views on the graphs like how you want them to be?

@ChrisRackauckas

Copy link
Copy Markdown
Member

Why are all of the success rates so low now?

image

BBO was basically 100%, now it's not even 50%. That's a massive regression, it obviously cannot be merged as-is.

@AdityaPandeyCN

AdityaPandeyCN commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

I increased the dimension of the BBO test functions to 10 from 3, which has increased the difficulty and caused the regression.
I will do some tuning and runs to get the best of pso before we can merge

@ChrisRackauckas

Copy link
Copy Markdown
Member

No please follow the procedure you would do with any other PR and isolate the changes. Do one PR that is just adding the new methods. Then one PR that is increasing the difficulty. Step by step changes where provenance is clear.

@AdityaPandeyCN

AdityaPandeyCN commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

Apologies, have reset to the relevant commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants