Updating Benchmark Sets

[AdityaPandeyCN]

Checklist

• Appropriate tests were added
• Any code changes were done in a way that does not break public API
• All documentation related to code changes were updated
• The new code follows the
  contributor guidelines, in particular the SciML Style Guide and
  COLPRAC.
• Any new documentation only uses public API

Additional context

This PR adds the recently implemented SciPy global optimization algorithms from OptimizationSciPy.jl (#927) to the black-box global optimizer benchmarks.

[ChrisRackauckas]

It's going to need to wait for the registration to finish JuliaRegistries/General#133534 JuliaRegistries/General#133535

Also could you add the new PyCMA one which just merged around the same time? SciML/Optimization.jl#933

[AdityaPandeyCN]

Sure

[ChrisRackauckas]

https://buildkite.com/julialang/scimlbenchmarks-dot-jl/builds/3473#0197b061-b0dd-4ffe-8fc5-2e2d200a57c8

@mxpoch it looks like PyCMA is giving a segfault and having some IO issues as part of the benchmark. Does PyCMA have to do IO in order to give the results? Seems like it could just all be memory?

[AdityaPandeyCN]

Hi @ChrisRackauckas Do you think we should run the Python-based optimizers sequentially to avoid the errors, this maybe happening because of multithreaded Python calls in Julia?

[mxpoch]

@ChrisRackauckas Yep the default behaviour of PyCMA is to store logs in an 'outcmaes' folder. If you pass verb_log=0 as an argument it shouldn't do any writes.

[AdityaPandeyCN]

I have made changes to run the SciPy optimizers sequentially, I have tried to run this on my local machine but this became overwhelming for my machine:-

1. In one of the iteration the sphere function test ran successfully and moved to the ellipsoidal but I interrupted as it ran for hours, the sphere test function was the one which corrupted so maybe running sequentially can fix this.
2. I scaled down the iteration and they ran correctly but I don't think this verifies anything? as the error happened at approx 60 billion iterations(https://buildkite.com/julialang/scimlbenchmarks-dot-jl/builds/3474/steps/canvas?sid=0197b0e1-97d8-43ef-866b-8264b670d97c)
   Any suggestion @ChrisRackauckas

[ChrisRackauckas]

as the error happened at approx 60 billion iterations

That seems like it would be over the limit? Is the maxiters not being passed on?

[AdityaPandeyCN]

Are we referring to the run_length here?, if so than yes seems like the SciPy optimizers are ignoring it and we have to set an explicit maxiters here.

[ChrisRackauckas]

Yes run_length is maxiters https://github.com/jonathanBieler/BlackBoxOptimizationBenchmarking.jl/blob/3026c725f4303f05a0a558386c485e689428bff2/src/benchmark.jl#L100

[ChrisRackauckas]

It looks like it is handled though: https://github.com/SciML/Optimization.jl/blob/master/lib/OptimizationSciPy/src/OptimizationSciPy.jl#L331.

It might be best to try and just isolate one problem

[AdityaPandeyCN]

I think I have found the root cause in my ScipyBasinhopping wrapper (inner optimiser runs un-bounded). I’ve dropped Basinhopping from the current benchmark and running this on my machine if this works well I will push it and fix the wrapper code.

[ChrisRackauckas]

oh nice! Does the nested optimizer need something like inner vs outer iterations?

[AdityaPandeyCN]

Basinhopping counts hops with niter (outer loop), then calls a local minimizer by default L-BFGS-B that has its own gradient iterations. We already set niter, but we also need to pass something like minimizer_kwargs[:options]["maxiter"] = … so each inner L-BFGS-B run is limited, otherwise a single hop can spin for billions.
This is what I am understanding as of now for this

[AdityaPandeyCN]

Hello @ChrisRackauckas I commented out two of the global optimizers the ScipyBasinhopping( reason we are discussing here) and ScipyDualAnnealing(this was really slow on my computer) I ran the benchmark and this is the result https://github.com/AdityaPandeyCN/juliabenchmark/blob/main/GlobalOptimization/blackbox_global_optimizers.md

Please have a look, also note that the DualAnnealling one was really slow it didn't crash so commented it out... it ran till here
┌ Info: Chain(ScipyDualAnnealing → NelderMead) └ Sphere
┌ Info: Chain(ScipyDualAnnealing → NelderMead) └ Ellipsoidal
I pressed ctrl c after this because this was taking really long time.

[AdityaPandeyCN]

Hello @ChrisRackauckas Any updates on this, anything I can help with you?

[ChrisRackauckas]

The last run didn't build right, right? I just fixed the merge conflict and will see what happens when it runs again.

[ChrisRackauckas]

yeah Error: UndefVarError: ScipyDifferentialEvolution not defined

[ChrisRackauckas]

It seems that still fails?

[AdityaPandeyCN]

@ChrisRackauckas I am working on it again will update you with the corrections.

[ChrisRackauckas]

Okay thanks. Yeah it seems like it's really close but it's not clear why it can't find that method.

[AdityaPandeyCN]

@ChrisRackauckas The Python packages fail to load due to CondaPkg conflicts. I can add this to the build script(https://github.com/SciML/SciMLBenchmarks.jl/blob/master/.buildkite/build_benchmark.sh) to use system Python instead:

if [[ "${1}" == *GlobalOptimization* ]]; then
    pip3 install --user scipy numpy cma
    export JULIA_PYTHONCALL_EXE=$(which python3)
    export JULIA_CONDAPKG_BACKEND="Null"
fi

Should I go ahead with this approach?

[ChrisRackauckas]

What exactly is the CondaPkg conflicts? Seems like something useful to work through for users? Or is it something obscure?

[AdityaPandeyCN]

From what I'm seeing, libstdcxx-ng resolution is failing in pixi's multi-platform solve(I guess), which crashes IJulia's CondaPkg initialization and prevents Python-based optimizers from precompiling. I tested locally by bypassing CondaPkg entirely (using system Python), which worked.

[AdityaPandeyCN]

Hello @ChrisRackauckas , the build is running but the ScipyDualAnnealing is taking really long time at this point like for 3 hours.

[ChrisRackauckas]

Is it possible to just cut that one out?

[AdityaPandeyCN]

Yes, it should be but they are running unlike crashing in our previous runs. Current progress

Info: Chain(ScipyDualAnnealing → NelderMead)
└ 	 Sphere
┌ Info: Chain(ScipyDualAnnealing → NelderMead)
└ 	 Ellipsoidal
┌ Info: Chain(ScipyDualAnnealing → NelderMead)
└ 	 Rastrigin

[AdityaPandeyCN]

@ChrisRackauckas, The log

2025-11-18 19:20:44 IST
┌ Info: Chain(ScipyDualAnnealing → NelderMead)
2025-11-18 19:20:44 IST
└ 	 Sphere
2025-11-18 22:40:08 IST
┌ Info: Chain(ScipyDualAnnealing → NelderMead)
2025-11-18 22:40:08 IST
└ 	 Ellipsoidal
2025-11-19 06:57:36 IST
┌ Info: Chain(ScipyDualAnnealing → NelderMead)
2025-11-19 06:57:36 IST
└ 	 Rastrigin
2025-11-19 13:16:46 IST
┌ Info: Chain(ScipyDualAnnealing → NelderMead)
2025-11-19 13:16:46 IST
└ 	 Buche-Rastrigin
2025-11-19 23:41:20 IST
┌ Info: Chain(ScipyDualAnnealing → NelderMead)
2025-11-19 23:41:20 IST
└ 	 Linear Slope
2025-11-20 02:13:54 IST
┌ Info: Chain(ScipyDualAnnealing → NelderMead)
2025-11-20 02:13:54 IST
└ 	 Attractive Sector

The duration is really large, how should we move ahead like stop it or let it run?

[ChrisRackauckas]

If you just push a commit it'll go to the next one.

[AdityaPandeyCN]

@ChrisRackauckas I disabled it and we have benhmark results now.

[AdityaPandeyCN]

Hello @ChrisRackauckas Can you have a look and tell what needs to be done more?

[ChrisRackauckas]

oh I missed that this worked, nice!