Fix and update AdaptiveSDE benchmarks (#126)

[jitendravjh]

Fixes #126. This is the AdaptiveSDE small-grant work (declared in SciML/sciml.ai#242).

The benchmarks weren't building. The main blocker was the manual Distributed parallelism (addprocs/@everywhere/ParallelDataTransfer.sendto), which Weave can't run, plus some bitrot against the current interfaces. Both files now use EnsembleProblem + EnsembleThreads() and weave cleanly.

A few things changed along the way. Dropped DiffEqMonteCarlo (the Monte Carlo runs go through EnsembleProblem now) and the unused Distributed/ParallelDataTransfer. Renamed calculate_monte_errors to calculate_ensemble_errors (fields unchanged) and added DiffEqBase as a dep. Fixed the keyword drift too, maxIters became maxiters, and I kept SciMLLogging.None() since a Bool verbose isn't accepted anymore. Also moved to ctx.sim_id for per-trajectory seeding and sol.u[i] for indexing. And there was a small existing bug in qmaxDetermination where the "other problems" loop was re-running problem 1 instead of probs[k], that's fixed now. Project.toml/Manifest.toml are updated to match.

I validated both files by weaving them locally at reduced and medium scale (Ns=[5,5,5], 200 trajectories, full qmax sweep). No errors, and the plots regenerate fine.

One thing to flag, the qmax-on-Oval2 runs all hit DtLessThanMin across the whole sweep (SRIW1 at abstol=2^-15 on Oval2). This matches the original solve settings so the port is faithful, but that section won't show qmax differentiation as-is, happy to relax the tolerances there if you'd prefer.

[jitendravjh]

@ChrisRackauckas the CI run for benchmarks/AdaptiveSDE timed out at 24h. It's not a build error, both files weave fine locally at reduced scale. It's just the workload, AdaptiveEfficiencyTests runs 1000 trajectories × up to 23 tolerance levels (down to abstol=2^-24) × 3 RSwM algorithms × 3 problems, so it doesn't finish in 24h even with EnsembleThreads() across all cores. The scale is inherited from the original, and the port itself is faster than the old addprocs(2) version.

Could you let me know what runtime limit or parameter scale you'd want here, trajectories, length of the Ns tolerance sweep, and num_runs for qmax? I'll dial it down to something that completes. Or if you'd rather keep full scale on dedicated infra with a longer time limit, that works too.

Let me know if you have any thoughts, happy to iterate further on your suggestions.

[AdityaPandeyCN]

Did the runner actually pick up this benchmark? Iterations are never a problem as they are expected to run for longer duration. If it timed out because no runner picked it(which looks like the case here) you will have to wait

[jitendravjh]

Thanks @AdityaPandeyCN, so it's runner availability, not the workload. I'll keep the parameters at full scale then. Happy to wait for a runner to pick it up.

[ChrisRackauckas]

k was 1. The number of Adaptive Fails is 9999. Elapsed time was 8.332477595
k was 2. The number of Adaptive Fails is 9999. Elapsed time was 2.586814747
k was 3. The number of Adaptive Fails is 9999. Elapsed time was 4.142193076
k was 4. The number of Adaptive Fails is 9999. Elapsed time was 4.583869417
k was 5. The number of Adaptive Fails is 9999. Elapsed time was 4.014884734
k was 6. The number of Adaptive Fails is 9999. Elapsed time was 1.686423312
k was 7. The number of Adaptive Fails is 9999. Elapsed time was 5.934806418
k was 8. The number of Adaptive Fails is 9999. Elapsed time was 2.646599562

[jitendravjh]

Thanks. I looked into it and the fails are flat across qmax in every configuration I tried, so the section can't determine a qmax as written. Loosening the tolerance or largeFluctuations=false makes it solvable (0 fails), but fails and times stay flat across all 8 qmax. Stiff solvers (ImplicitEM, ImplicitRKMil, STrapezoid) solve it cleanly with times still flat, even at 2^-16.

How do you want to handle it: drop the oval2 qmax section, swap in a problem or solver where qmax actually matters, or change the metric?

[ChrisRackauckas]

It should be in the new controller setup arguments.

[jitendravjh]

Done, routed qmax through the controller in both sweeps.

On the Oval2 test though, it still fails ~100% across all qmax. Should I leave it as-is, or swap it for a solver/tolerance that completes?

[ChrisRackauckas]

Why does it fail? If you remove the try/catch stuff, what is it saying?

[jitendravjh]

No try/catch, just verbose = SciMLLogging.None().
dt(2.220446049250313e-16) <= dtmin(2.220446049250313e-16) at t=0.006961747059462434. Aborting. There is either an error in your model specification or the true solution is unstable.

[ChrisRackauckas]

It shouldn't be failing like that?

[jitendravjh]

Right, it's the solver not the problem. SRIW1 collapses dt to eps on large-fluctuation oval2 at any tolerance (even 0.1), and fixed dt needs dt <= 1/2^16 to stay stable, so it's stability not accuracy. SOSRI and SRA3 solve it fine, and with SOSRI the qmax sweep actually differentiates: 0 fails at qmax=1.03 up to 43/50 at qmax=5.

[jitendravjh]

Swapped the Oval2 test to SOSRI, qmax curve is clean now.

[ChrisRackauckas]

There are a bunch of adaptive failures. See an old version to see what you should be getting when it's corrected.

https://github.com/SciML/SciMLBenchmarks.jl/blob/70f4f23bc27fb0408510f565397d624e88cb21d4/pdf/AdaptiveSDE/qmaxDetermination.pdf

Also, the figures of the other one need to transpose the labels to fix the plot legend.

[jitendravjh]

Transposed the labels on AdaptiveEfficiencyTests, the legend maps correctly now. Moved the Oval2 qmax test to the reference tolerance (2^-15/2^-10), which cuts the failures down a lot but not to zero. From what I can tell this is an upstream regression, SRIW1 solved oval2 with 0 fails in 2019 but fails 100% now, and SOSRI is the closest current solver.
Any idea what changed?

[ChrisRackauckas]

The time stepper is probably very different from before, so it might need to be tuned.

[jitendravjh]

That did it. Dropping the controller safety factor to gamma=0.7 gets 0 fails across the whole qmax sweep now, matching the reference.

[jitendravjh]

Hi @ChrisRackauckas, anything else you'd like on this one?
AdaptiveSDE CI job keeps getting cancelled at 24h waiting for a runner, so it hasn't actually run on CI yet.

[ChrisRackauckas]

It ran and it stalled for 4 days before it cancelled the run:

  127 dependencies successfully precompiled in 148 seconds. 245 already precompiled.
┌ Warning: The project dependencies or compat requirements have changed since the manifest was last resolved.
│ It is recommended to `Pkg.resolve()` or consider `Pkg.update()` if necessary.
└ @ Pkg.API ~/github-runners/amdci3-1/_work/_tool/julia/1.12.6/x64/share/julia/stdlib/v1.12/Pkg/src/API.jl:1322
Building Script
[ Info: Tangled to /home/crackauc/github-runners/amdci3-1/_work/SciMLBenchmarks.jl/SciMLBenchmarks.jl/script/AdaptiveSDE/AdaptiveEfficiencyTests.jl
Building Github Markdown
┌ Info: Weaving chunk 1 from line 6
└   progress = 0.0
┌ Info: Weaving chunk 2 from line 54
└   progress = 0.14285714285714285
┌ Warning: Assignment to `prob` in soft scope is ambiguous because a global variable by the same name exists: `prob` will be treated as a new local. Disambiguate by using `local prob` to suppress this warning or `global prob` to assign to the existing global variable.
└ @ ~/github-runners/amdci3-1/_work/SciMLBenchmarks.jl/SciMLBenchmarks.jl/benchmarks/AdaptiveSDE/AdaptiveEfficiencyTests.jmd:15
┌ Info: Weaving chunk 3 from line 124
└   progress = 0.2857142857142857
GKS: cannot open display - headless operation mode active
┌ Info: Weaving chunk 4 from line 157
└   progress = 0.42857142857142855
┌ Info: Weaving chunk 5 from line 162
└   progress = 0.5714285714285714
┌ Info: Weaving chunk 6 from line 168
└   progress = 0.7142857142857143
┌ Info: Weaving chunk 7 from line 172
└   progress = 0.8571428571428571
┌ Info: Weaved all chunks
└   progress = 1
[ Info: Weaved to /home/crackauc/github-runners/amdci3-1/_work/SciMLBenchmarks.jl/SciMLBenchmarks.jl/markdown/AdaptiveSDE/AdaptiveEfficiencyTests.md
[ Info: Weaving benchmarks/AdaptiveSDE/qmaxDetermination.jmd
┌ Info: Instantiating
└   folder = "benchmarks/AdaptiveSDE"
  Activating project at `~/github-runners/amdci3-1/_work/SciMLBenchmarks.jl/SciMLBenchmarks.jl/benchmarks/AdaptiveSDE`
┌ Warning: The project dependencies or compat requirements have changed since the manifest was last resolved.
│ It is recommended to `Pkg.resolve()` or consider `Pkg.update()` if necessary.
└ @ Pkg.API ~/github-runners/amdci3-1/_work/_tool/julia/1.12.6/x64/share/julia/stdlib/v1.12/Pkg/src/API.jl:1322
    Building Conda ─→ `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/8f06b0cfa4c514c7b9546756dbae91fcfbc92dc9/build.log`
    Building IJulia → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/102656c4efc9737f892e1bca7e66ae374c650740/build.log`
┌ Warning: The project dependencies or compat requirements have changed since the manifest was last resolved.
│ It is recommended to `Pkg.resolve()` or consider `Pkg.update()` if necessary.
└ @ Pkg.API ~/github-runners/amdci3-1/_work/_tool/julia/1.12.6/x64/share/julia/stdlib/v1.12/Pkg/src/API.jl:1322
┌ Warning: The project dependencies or compat requirements have changed since the manifest was last resolved.
│ It is recommended to `Pkg.resolve()` or consider `Pkg.update()` if necessary.
└ @ Pkg.API ~/github-runners/amdci3-1/_work/_tool/julia/1.12.6/x64/share/julia/stdlib/v1.12/Pkg/src/API.jl:1322
Building Script
[ Info: Tangled to /home/crackauc/github-runners/amdci3-1/_work/SciMLBenchmarks.jl/SciMLBenchmarks.jl/script/AdaptiveSDE/qmaxDetermination.jl
Building Github Markdown
┌ Info: Weaving chunk 1 from line 6
└   progress = 0.0
┌ Info: Weaving chunk 2 from line 55
└   progress = 0.25
┌ Warning: Assignment to `sol` in soft scope is ambiguous because a global variable by the same name exists: `sol` will be treated as a new local. Disambiguate by using `local sol` to suppress this warning or `global sol` to assign to the existing global variable.
└ @ timing.jl:461
Error: The operation was canceled.

[ChrisRackauckas]

qmaxdetermination shouldn't take 4 days?

[jitendravjh]

Found it. 0 fails, ~100k steps each, runs in minutes now.

qmax comes out flat though, steps and times are the same across all 8 values, so it can't determine a qmax like the old reference. Looks like the stepper change you mentioned. Leave it as-is, or is there a solver you'd expect to still show qmax sensitivity here?

@ChrisRackauckas

[ChrisRackauckas]

That's not a solution, force_dtmin shouldn't really ever be true.

[jitendravjh]

Dug into this properly. Looks like an over-refinement regression in StochasticDiffEq's adaptive SRK error control, not the benchmark. At the reference tolerance SRIW1 collapses to dtmin even on the easy paths, and SOSRI runs about 130x below the stability limit. SROCKEM holds the stability-limit step and solves oval2 in ~250 steps with 0 fails.

The 0-fail solvers don't respond to qmax though, so they can't do the determination. Tune the stepping upstream, or swap the solver here?

[jitendravjh]

Dropped force_dtmin and switched oval2 to SROCKEM. Solves at the reference tolerance in ~257 steps, 0 fails across the whole qmax sweep, runs in seconds.

qmax's still flat on it like the other current solvers, so getting that back means tuning the stepper upstream. Let me know which way you'd rather go.