[BUG] Solve() aborts the process (OMP kmp_alloc assertion) when each solve runs on a freshly created thread

[mal84emma]

Note

The investigation and this report were prepared by an AI agent (Claude) on behalf of the reporter.

Describe the bug

When each Solve() call is made from a newly created Python thread, cuOpt aborts the process after a small number of sequential solves:

OMP: Error #13: Assertion failure at kmp_alloc.cpp(2725).

(exit 134), or a bare SIGSEGV with no message at all (exit 139). There is no Python exception and no traceback — the interpreter dies. Running the identical solves on the main thread, or on a single long-lived worker thread, is completely clean.

This looks like per-thread state in the bundled LLVM OpenMP runtime (libomp-8fe85495.so, a NEEDED of libcuopt.so, resolving to libcuopt_cu12.libs/libomp-8fe85495.so) that is not released when the thread that entered the solver exits, so each new solver-entering thread leaks a thread record until the runtime's allocator invariant breaks.

Both MILP and LP are affected. MILP merely reaches the threshold fastest — consistently on the 2nd solve here, versus the 5th–13th for LP. The failing signature is identical in both cases.

This is a hard, uncatchable process abort, and it is triggered by an entirely ordinary pattern: thread-per-request services, and worker-thread wrappers used to keep the calling thread interruptible.

Steps/Code to reproduce bug

Self-contained; only cuopt and numpy are needed. A small knapsack (20 variables, 3 <= rows) that is trivially feasible (x = 0) and bounded, solved N times sequentially, with a --mode switch controlling which thread each solve runs on.

#!/usr/bin/env python3
"""Minimal reproduction: cuOpt Solve() aborts the process when each solve runs
on a freshly created Python thread."""

import argparse
import queue
import sys
import threading

import numpy as np

from cuopt.linear_programming import DataModel, Solve, SolverSettings
from cuopt.linear_programming.solver.solver_parameters import CUOPT_TIME_LIMIT


def build_model(integer=True, n=20, seed=0):
    """Small, definitely feasible and bounded knapsack.

    maximize sum_i v_i x_i  s.t.  sum_i w_ij x_i <= cap_j  (j = 0..2),
    0 <= x_i <= 1, x_i integer (or continuous with --lp).
    """
    rng = np.random.default_rng(seed)
    m = 3
    w = rng.integers(1, 10, size=(m, n)).astype(np.float64)
    v = rng.integers(1, 20, size=n).astype(np.float64)
    cap = 0.5 * w.sum(axis=1)

    model = DataModel()
    offsets = np.arange(m + 1, dtype=np.int32) * n
    indices = np.tile(np.arange(n, dtype=np.int32), m)
    model.set_csr_constraint_matrix(w.reshape(-1), indices, offsets)
    model.set_row_types(np.array(["L"] * m, dtype=object))
    model.set_constraint_bounds(cap)
    model.set_objective_coefficients(v)
    model.set_maximize(True)
    model.set_variable_lower_bounds(np.zeros(n))
    model.set_variable_upper_bounds(np.ones(n))
    model.set_variable_types(np.array(["I" if integer else "C"] * n, dtype=object))
    return model


def solve_once(i, integer, out):
    model = build_model(integer=integer, seed=i)
    settings = SolverSettings()
    settings.set_parameter(CUOPT_TIME_LIMIT, 5.0)
    solution = Solve(model, settings)
    out.append((i, str(solution.get_termination_reason()),
                float(solution.get_primal_objective())))


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--mode", required=True,
                    choices=["main", "fresh-thread", "reused-thread"])
    ap.add_argument("--n-solves", type=int, default=10)
    ap.add_argument("--lp", action="store_true",
                    help="solve a pure LP (continuous variables) instead of a MILP")
    args = ap.parse_args()

    integer = not args.lp
    kind = "MILP" if integer else "LP"
    print(f"mode={args.mode} kind={kind} n_solves={args.n_solves}", flush=True)

    def run(i):
        out = []
        if args.mode == "main":
            solve_once(i, integer, out)
        elif args.mode == "fresh-thread":
            # A brand new OS thread for every single solve.
            t = threading.Thread(target=solve_once, args=(i, integer, out))
            t.start()
            t.join()
        return out

    if args.mode == "reused-thread":
        # One long-lived worker thread handles every solve.
        jobs, results = queue.Queue(), []

        def worker():
            while True:
                i = jobs.get()
                if i is None:
                    return
                solve_once(i, integer, results)
                print(f"  solve {i + 1}/{args.n_solves} -> {results[-1][1]}", flush=True)

        t = threading.Thread(target=worker)
        t.start()
        for i in range(args.n_solves):
            jobs.put(i)
        jobs.put(None)
        t.join()
        done = len(results)
    else:
        done = 0
        for i in range(args.n_solves):
            out = run(i)
            if not out:
                print(f"  solve {i + 1}/{args.n_solves} -> NO RESULT", flush=True)
                sys.exit(1)
            done += 1
            print(f"  solve {i + 1}/{args.n_solves} -> {out[-1][1]}", flush=True)

    print(f"ALL {done} SOLVES COMPLETED", flush=True)


if __name__ == "__main__":
    main()

Run each in its own process, so the abort is observable rather than killing a parent:

python repro.py --mode main          --n-solves 10        # MILP, main thread          -> clean
python repro.py --mode reused-thread --n-solves 10        # MILP, one worker thread    -> clean
python repro.py --mode fresh-thread  --n-solves 10        # MILP, thread per solve     -> ABORTS
python repro.py --mode fresh-thread  --n-solves 30 --lp   # LP,   thread per solve     -> ABORTS

Observed output in fresh-thread mode — the process dies mid-solve with no Python-level error:

Running cuOpt presolve
Running probing cache with 7 tasks
Assertion failure at kmp_alloc.cpp(2725): ((kmp_mem_descr_t *)((char *)next - sizeof(kmp_mem_descr_t))) ->size_allocated + 1 == ((kmp_mem_descr_t *)((char *)tail - sizeof(kmp_mem_descr_t))) ->size_allocated.
OMP: Error #13: Assertion failure at kmp_alloc.cpp(2725).
OMP: Hint Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see https://github.com/llvm/llvm-project/issues/.

Key evidence — where the solve runs is what decides it

Every row is one process; "solves completed" counts solves that finished before the process died.

Problem	Solve thread	Runs	Result
MILP	main thread	1	10/10 clean, exit 0
MILP	one reused worker thread	1	10/10 clean, exit 0
MILP	fresh thread per solve	6	6/6 aborted, always after exactly 1 completed solve (dies during solve 2) — exit 134 (5x), exit 139 (1x)
LP	main thread	1	30/30 clean, exit 0
LP	one reused worker thread	1	30/30 clean, exit 0
LP	fresh thread per solve	4	4/4 aborted after 5, 8, 10 and 13 completed solves — exit 134 (2x), exit 139 (2x)

Two notes on the LP row, since it is easy to misread this as MIP-specific:

• An LP fresh-thread run limited to 10 solves finished cleanly 10/10. Capped at small N, the LP path looks unaffected; at N=30 it fails every time. The LP path has a higher threshold, not immunity.
• The LP runs take a genuine LP path — the log reports 20 variables (0 integers), PSLP presolve and concurrent dual simplex / PDLP / barrier, with no branch-and-bound anywhere.

Expected behavior

Every mode should print ALL <n> SOLVES COMPLETED and exit 0. Repeatedly entering the solver from short-lived threads should not accumulate unbounded per-thread OpenMP state, and certainly should not abort the process.

Environment details (please complete the following information):

• Environment location: Cloud (Microsoft Azure VM), bare-metal Python virtualenv — no container
• Method of cuOpt install: pip, from public PyPI (cuopt-cu12==26.8.0)

cuOpt	cuopt-cu12 / libcuopt-cu12 26.8.0 (cuopt.__version__ = 26.08.00, git hash 400863c)
Latest on PyPI	26.8.0 — the newest published release is the one tested
OpenMP runtime	bundled libomp-8fe85495.so (LLVM libomp) in libcuopt_cu12.libs, a NEEDED of libcuopt.so
GPU / driver	Tesla T4, driver 535.274.02
CUDA	cuOpt banner reports CUDA 12.9; nvidia-cuda-runtime-cu12 12.9.79
CPU	AMD EPYC 7V12, 8 physical / 8 logical threads
OS / kernel	Ubuntu 22.04.5 LTS, 6.8.0-1044-azure
Python	3.11.15

Additional context

Workaround: never let a thread that has entered Solve() exit and be replaced by a new one. Either solve on the main thread, or route every solve through a single long-lived worker thread. Both were clean across every run above, including LP at 30 solves.

Found while integrating cuOpt as a linopy backend, where solves were dispatched on a worker thread per call to keep the calling thread interruptible.

Related but, as far as I can tell, distinct existing issues:

• [BUG] MIP SIGSEGV - Multiple OpenMP runtime conflict #1219 (open, "MIP SIGSEGV - Multiple OpenMP runtime conflict") — a SIGSEGV inside libgomp during post-solve teardown of one large MIP, attributed to several libgomp copies in one process. This report instead involves the bundled LLVM libomp (kmp_* symbols; libgomp has no kmp_alloc), needs no large model, and depends entirely on which thread calls Solve().
• [BUG] MIP solver thread count degrades on 2nd+ solve due to omp_get_max_threads() contamination #900 (closed, "MIP solver thread count degrades on 2nd+ solve due to omp_get_max_threads() contamination") — also about OpenMP state surviving across sequential solves, but its symptom is reduced parallelism and a slowdown, not a crash.

One observation offered only as a possible lead, not a diagnosis: PR #1099 migrated most of the MIP solver from std::thread to OpenMP tasking, and notes that "only a single parallel region [is] created at the beginning of the solver, so it can be shared across the MIP solver." A parallel region opened once per solve entry, on whichever thread calls in, would be consistent with the MILP path exhausting the per-thread bookkeeping several times faster than the LP path does — but I have not traced the allocation to confirm that.

Full crash log — MILP, fresh-thread mode (tail)

New solution from early primal heuristics (CPUFJ). Objective +1.190000e+02. Time 0.01
New solution from early primal heuristics (CPUFJ). Objective +1.260000e+02. Time 0.01
New solution from early primal heuristics (CPUFJ). Objective +1.280000e+02. Time 0.01

Running Papilo presolve (git hash 32b3a87d)
Presolve status: did not result in any changes
Presolve removed: 0 constraints, 0 variables, 0 nonzeros
Presolved problem: 3 constraints, 20 variables (20 integer), 60 nonzeros
Papilo presolve time: 0.02
Objective offset -0.000000 scaling_factor -1.000000
Model fingerprint: 0x388ac148

Running cuOpt presolve
Running probing cache with 7 tasks
Assertion failure at kmp_alloc.cpp(2725): ((kmp_mem_descr_t *)((char *)next - sizeof(kmp_mem_descr_t))) ->size_allocated + 1 == ((kmp_mem_descr_t *)((char *)tail - sizeof(kmp_mem_descr_t))) ->size_allocated.
OMP: Error #13: Assertion failure at kmp_alloc.cpp(2725).
OMP: Hint Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see https://github.com/llvm/llvm-project/issues/.

Full crash log — LP, fresh-thread mode (tail), showing the LP-only solve path

Solving a problem with 3 constraints, 20 variables (0 integers), and 60 nonzeros
...
Using PSLP presolver
PSLP Presolved problem: 3 constraints, 20 variables, 60 non-zeros
PSLP presolve time: 0.00s
Objective offset -0.000000 scaling_factor -1.000000
Running concurrent (showing only PDLP log)

Dual simplex finished in 0.00 seconds
Barrier finished in 0.01 seconds
Assertion failure at kmp_alloc.cpp(2725): ((kmp_mem_descr_t *)((char *)next - sizeof(kmp_mem_descr_t))) ->size_allocated + 1 == ((kmp_mem_descr_t *)((char *)tail - sizeof(kmp_mem_descr_t))) ->size_allocated.
OMP: Error #13: Assertion failure at kmp_alloc.cpp(2725).

First-solve banner (version / device / build identification)

Setting parameter time_limit to 5.000000e+00
cuOpt version: 26.8.0, git hash: 400863c1, host arch: x86_64, device archs: 70-real,75-real,80-real,86-real,90a-real,100f-real,120a-real,120
CPU: AMD EPYC 7V12 64-Core Processor, threads (physical/logical): 8/8, RAM (available/total): 49.73 / 54.92 GiB
CUDA 12.9, device: Tesla T4 (ID 0), VRAM: 15.57 GiB

[mal84emma]

Note

The following content was generated by AI (investigation supervised by the repository maintainer's project lead).

Graver manifestation of the same runtime: glibc main-arena large-bin metadata corruption. Continued work on the same integration has produced a native backtrace showing that the process heap itself is damaged — not just libomp's internal allocator. The Error #13 abort reported above is the loud case; the quiet case is that any long-lived process that has solved with cuOpt can later SIGSEGV in completely unrelated code, at an allocation cuOpt never made, arbitrarily long after every cuOpt solve has returned. Same environment as the original report (cuopt-cu12 26.8.0 / cuopt.__version__ 26.08.00 git hash 400863c1, Tesla T4, driver 535.274.02, Ubuntu 22.04.5, Python 3.11.15) — and here every solve ran on one long-lived worker thread, i.e. the workaround recommended in the original report. That workaround stops the kmp_alloc assertion; it does not stop this.

1. The native backtrace — _int_malloc walking a large-bin skip list through a NULL chunk

Full pytest suite under gdb --batch. The faulting allocation is a 1 MiB malloc made by CPLEX, a library entirely unrelated to cuOpt, in the main arena:

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007ffff7ca45c2 in _int_malloc (av=av@entry=0x7ffff7e1ac80 <main_arena>, bytes=bytes@entry=1048592) at ./malloc/malloc.c:4133

#0  0x00007ffff7ca45c2 in _int_malloc (av=av@entry=0x7ffff7e1ac80 <main_arena>, bytes=bytes@entry=1048592) at ./malloc/malloc.c:4133
#1  0x00007ffff7ca5139 in __GI___libc_malloc (bytes=1048592) at ./malloc/malloc.c:3329
#2  0x00007ffc9070de3c in _ce2c849dea36cd50f814ed0fad2731f8 () from .../cplex/_internal/libcplex2220.so
...
#10 0x00007ffd1ac3e921 in _wrap_CPXXreadcopyprob () from .../cplex/_internal/py311_cplex2220.so
#11 0x000055555589647f in cfunction_call ()

Registers, siginfo and the faulting instruction:

rip            0x7ffff7ca45c2      0x7ffff7ca45c2 <_int_malloc+3090>
rax            0x0                 0
rdi            0x0                 0
rsi            0x55556c33b988      93825375910280

$1 = { si_signo = 11, si_errno = 0, si_code = 1, ...
       _sigfault = {si_addr = 0x20, _addr_lsb = 0, _addr_bnd = {_lower = 0x0, _upper = 0x0}} ... }

=> 0x7ffff7ca45c2 <_int_malloc+3090>:	cmp    %rsi,0x20(%rax)
   0x7ffff7ca45c6 <_int_malloc+3094>:	jne    0x7ffff7ca46f2 <_int_malloc+3394>
   0x7ffff7ca45cc <_int_malloc+3100>:	mov    %rdx,0x28(%rsi)
   0x7ffff7ca45d0 <_int_malloc+3104>:	mov    0x28(%rdx),%rax

Reading: 0x20 and 0x28 are the fd_nextsize / bk_nextsize offsets of an x86-64 malloc_chunk. _int_malloc is traversing the main arena's large-bin doubly-linked skip list to service the 1 MiB request; rax = 0, so the chunk pointer it just loaded out of a bin is NULL, and si_addr = 0x20 is exactly NULL->fd_nextsize. This is glibc free-list metadata corruption in the main arena — not a stale Python object pointer, not a numpy/xarray defect, not the payload of any one allocation.

It also explains the shape of every other crash we have seen: the corruption is silent until someone walks the broken bin, and who that someone is depends only on allocation history. The victim is never cuOpt.

At the moment of the fault, 76 threads were live, 7 of them cuOpt's LLVM OpenMP team still parked inside libomp-8fe85495.so, long after the last cuOpt solve had completed and its DataModel had been released:

Thread 187 (... "python"):
#5  __kmp_suspend_64<false, true>(int, kmp_flag_64<false, true>*)   from .../libcuopt_cu12.libs/libomp-8fe85495.so
#6  kmp_flag_64<false, true>::wait(kmp_info*, int, void*)           from .../libomp-8fe85495.so
#7  __kmp_hyper_barrier_release(barrier_type, kmp_info*, int, int, int, void*) from .../libomp-8fe85495.so
#8  __kmp_fork_barrier(int, int)                                    from .../libomp-8fe85495.so
#9  __kmp_launch_thread ()                                          from .../libomp-8fe85495.so
#10 __kmp_launch_worker(void*)                                      from .../libomp-8fe85495.so

That is the same runtime whose own allocator asserted in kmp_alloc.cpp(2725) in the original report. A thread-caching allocator inside libomp writing through stale bin pointers into the shared glibc main arena is the one mechanism that fits both symptoms.

Full faulting-thread backtrace, thread-name census, and a second crash reaching the Python object graph

Faulting thread, all frames (logs/b21-diag-gdb-run2.txt):

#0  0x00007ffff7ca45c2 in _int_malloc (av=av@entry=0x7ffff7e1ac80 <main_arena>, bytes=bytes@entry=1048592) at ./malloc/malloc.c:4133
#1  0x00007ffff7ca5139 in __GI___libc_malloc (bytes=1048592) at ./malloc/malloc.c:3329
#2  0x00007ffc9070de3c in _ce2c849dea36cd50f814ed0fad2731f8 () from .../cplex/_internal/libcplex2220.so
#3  0x00007ffc90303bf8 in _28525deb8bddd46a623fb07e13979222 () from .../cplex/_internal/libcplex2220.so
#4  0x00007ffc90709269 in _19fd60bf6822670b5a3fbb6a01e41bb8 () from .../cplex/_internal/libcplex2220.so
#5  0x00007ffc90707216 in _50eb47d28184c02d7fd6eccda7757bd4 () from .../cplex/_internal/libcplex2220.so
#6  0x00007ffc90704421 in _252602275c785a191efbcd67ab980d9a () from .../cplex/_internal/libcplex2220.so
#7  0x00007ffc90704224 in _c9dd16a810fc381a8d2d5e2ddb362ae4 () from .../cplex/_internal/libcplex2220.so
#8  0x00007ffc906a35cb in _23c63df4cd545d77dcef4600df93a499 () from .../cplex/_internal/libcplex2220.so
#9  0x00007ffc9077f274 in _252b9ead8cde40286f58b7e006dc4493 () from .../cplex/_internal/libcplex2220.so
#10 0x00007ffd1ac3e921 in _wrap_CPXXreadcopyprob () from .../cplex/_internal/py311_cplex2220.so
#11 0x000055555589647f in cfunction_call ()
#12 0x000055555591808f in _PyEval_EvalFrameDefault ()
...

Thread-name census at crash time — 76 threads total; the 7 __kmp_launch_worker frames are all within the 42 threads named python:

     42 python            (7 of them parked inside libomp, 6 frames each)
     10 tokio-rt-worker   (polars)
      8 async-executor-
      5 jemalloc_bg_thd   (polars' bundled jemalloc)
      2 cuda-EvtHandlr
      9 polars-*

A separate gdb run, this one with MALLOC_CHECK_=3 MALLOC_PERTURB_=165 LD_PRELOAD=/lib/x86_64-linux-gnu/libc_malloc_debug.so.0, crashed with glibc printing no diagnostic whatsoever (grep -cE "malloc\(\)|free\(\)|corrupted|glibc detected" → 0). It shows the damage also reaches the CPython object graph — a live container holding a pointer that is no longer a valid object:

Thread 91 "python" received signal SIGSEGV, Segmentation fault.
0x000055555595e5a4 in visit_decref ()

#0  0x000055555595e5a4 in visit_decref ()
#1  0x0000555555890962 in dict_traverse ()
#2  0x000055555595cf94 in gc_collect_main ()
#3  0x00005555558b212f in PyType_GenericAlloc ()
#4  0x00005555559d3e61 in dict_new ()
#5  0x00005555558ad705 in type_call ()
...
#24 0x0000555555b16d3a in thread_run ()
#25 0x0000555555a5a5c4 in pythread_wrapper ()
#26 0x00007ffff7c94ac3 in start_thread (...) at ./nptl/pthread_create.c:442

si_signo = 11, si_code = 1, _sigfault = {si_addr = 0xaa}

So: one process-wide heap corruption with two observable faces — broken glibc bin pointers, and dangling object pointers.

2. Statistical evidence — cuOpt activity is the cause, by bisect

Every cell below is a full pytest test/ --run-gpu run of our suite on one fixed commit; only the cuOpt-related selection and the diagnostic tooling differ. Roughly 100 in-process cuOpt solves (mixed LP and MILP) occur early in each run, followed by heavy allocation churn from ~8 700 unrelated tests.

unpinned baseline                                     4 crashes / 6 runs
everything pinned + SolverSettings on the worker      2 crashes / 3 runs
under gdb, no allocator flags                         1 crash   / 2 runs
under gdb + MALLOC_CHECK_=3 + MALLOC_PERTURB_=165     1 crash   / 1 run
------------------------------------------------------------------------
combined                                              8 crashes / 12 runs

versus, with all cuOpt activity removed from the same suite at the same commit (--ignore=test/test_cuopt.py -k "not cuopt", 116 tests deselected):

0 crashes / 3 runs

and the pass/fail counts in those cuOpt-free runs are identical to our pre-cuOpt baseline. Clean runs with cuOpt are also always baseline-identical (FAILED/ERROR symmetric difference vs baseline = 0 over 77 ids), so this is a pure crash-or-not effect, not a wrong-answer effect. Machine health was verified independently (GPU 43 °C idle, 36 GiB RAM free); no other workload on the box.

Six distinct victim sites, in five unrelated libraries, all after cuOpt solves had finished:

#	site	thread
1	SCIP m.optimize() (PySCIPOpt native call)	main
2	CPython GC decref of a non-cuOpt solver's native model, from __del__	main
3	numpy _get_ufunc_and_otypes listcomp under apply_ufunc (×2 runs)	main
4	xarray/numpy where under an expression merge (×2 runs)	main
5	glibc _int_malloc large-bin walk, from CPLEX CPXXreadcopyprob	main
6	CPython gc_collect_main → dict_traverse → visit_decref	worker thread

Verbatim faulthandler tracebacks for sites 1–4

Site 1 — SCIP optimize():

Current thread 0x00007821f6c38740 (most recent call first):
  File ".../linopy/solvers.py", line 2693 in _run_file
  File ".../linopy/solvers.py", line 990 in solve
  File ".../linopy/model.py", line 2142 in solve
  File ".../test/test_optimization.py", line 536 in test_default_settings_small_slices

Site 2 — GC-time decref of an unrelated solver's native model:

Current thread 0x000078e7136be740 (most recent call first):
  Garbage-collecting
  File ".../linopy/solvers.py", line 1259 in close
  File ".../linopy/solvers.py", line 1264 in __del__
  File ".../xarray/core/indexes.py", line 1735 in __init__
  File ".../xarray/core/dataset.py", line 1846 in xindexes
  File ".../xarray/core/dataset.py", line 1213 in _copy
  File ".../xarray/core/dataset.py", line 1184 in copy

Site 3 — numpy ufunc dispatch, identical in two separate runs:

Current thread 0x00007059d6418740 (most recent call first):
  File ".../numpy/lib/_function_base_impl.py", line 2555 in <listcomp>
  File ".../numpy/lib/_function_base_impl.py", line 2555 in _get_ufunc_and_otypes
  File ".../numpy/lib/_function_base_impl.py", line 2596 in _vectorize_call
  File ".../numpy/lib/_function_base_impl.py", line 2511 in _call_as_normal
  File ".../numpy/lib/_function_base_impl.py", line 2518 in __call__
  File ".../xarray/computation/apply_ufunc.py", line 820 in apply_variable_ufunc
  File ".../xarray/computation/apply_ufunc.py", line 312 in apply_dataarray_vfunc
  File ".../xarray/computation/apply_ufunc.py", line 1267 in apply_ufunc

Site 4 — xarray/numpy where, again identical in two separate runs:

Current thread 0x00007b021ef43740 (most recent call first):
  File ".../xarray/core/duck_array_ops.py", line 428 in where
  File ".../xarray/core/duck_array_ops.py", line 434 in where_method
  File ".../xarray/core/duck_array_ops.py", line 409 in sum_where
  File ".../xarray/computation/nanops.py", line 101 in nansum
  File ".../xarray/core/duck_array_ops.py", line 551 in f
  File ".../xarray/namedarray/core.py", line 928 in reduce

3. Ruled out — so nobody has to re-chase these

Four candidate caller-side explanations were tested and eliminated:

1. Caller buffer lifetimes are not the cause. A source audit of data_model_wrapper.pyx plus a measured probe: DataModel.set_* does not retain zero-copy views. type_cast ends in np_obj.astype(np.dtype(np_type)) — an unconditional copy — stored in a cdef attribute of the DataModel, and set_data_model_view() then takes get_data_ptr(self.get_<field>()), a pointer into the DataModel's own buffer. Measured for all nine arrays we pass: same buffer? False, owned True for the eight numeric fields, and mutating every caller array after handing it over leaves the DataModel's contents unchanged after caller mutation: True. variable_types is the sole aliasing setter, and it cannot dangle either — solver_wrapper.Solve replaces it with an astype("S1") copy before set_data_model_view() runs (that cast is also what makes the <const char*> element count correct: <U1 is 4 bytes/elem, S1 is 1). All data_model_view_t length/index invariants (len(lb)==len(ub)==len(vtypes)==len(c), len(cl)==len(cu)==len(offsets)-1, offsets[-1]==nnz, indices.max() < n_vars, C-contiguous, aligned, no views) hold across all seven model shapes exercised: INVARIANTS OK on every row. Nothing out of bounds is handed to cuOpt.

2. Thread affinity of the DataModel / SolverSettings lifecycle is not the cause. A harness pinned forever every argument of every DataModel.set_* call, plus every DataModel, SolverSettings and Solution for the entire session — nothing cuOpt-related was ever freed — and additionally constructed SolverSettings on the same persistent worker thread that runs Solve, so no cuOpt native object crossed a thread boundary at all. Result: 2 crashes / 3 runs, statistically indistinguishable from 4/6 unpinned. The mitigation measurably changes nothing.

3. It is not a payload overflow. PYTHONMALLOC=debug, MALLOC_CHECK_=3, MALLOC_PERTURB_=165 and LD_PRELOAD=libc_malloc_debug.so.0 are all silent — including on the run that crashed under them. 4 000 half-freed 496-byte heap canaries bracketing every phase of a dedicated standalone reproducer were never touched, and CPython's guard bytes never fired. Consistent with the backtrace: the damage is to glibc's own bin pointers, not to any block's payload, so nothing these tools watch is ever disturbed.

4. Two vendored OpenMP runtimes coexist in the process — offered as an environmental fact, not a diagnosis. import cuopt.linear_programming alone brings in both, neither able to dedupe with the other because both have auditwheel-mangled private SONAMEs:

   .../libraft_cu12.libs/libgomp-855c301a.so.1.0.0     <- DT_NEEDED of libraft.so   (GNU)
   .../libcuopt_cu12.libs/libomp-8fe85495.so           <- DT_NEEDED of libcuopt.so  (LLVM)
   

   The LLVM one ships the full GNU shim (256 GOMP_* entry points, omp_*@OMP_1.0, version nodes GOMP_1.0…GOMP_5.0, 33 dynamic symbols in common with libgomp). We checked for interposition and it is not happening — LD_DEBUG=bindings shows every call staying inside its own runtime (libcuopt.so -> libomp-8fe85495.so for GOMP_parallel, omp_set_lock, …; libgomp-855c301a.so -> itself). So this is not symbol confusion. But it does mean two independent OMP runtimes with two thread pools and two sets of TLS destructors in one process, which may be relevant context for Fix libomp ABI bug #1593's GOMP_task compatibility-layer findings.

4. Reproduction

Not reduced to a small standalone script, and we want to be straight about that. Three standalone reproducers all came back clean under every allocator-debugging flag listed above — one at 156 cuOpt solves plus numpy churn; one at 140 cuOpt solves plus 3 600 non-cuOpt native solves (highs/scip/copt/xpress/cplex/gurobi, file and direct, models closed) plus 5 live thread-pool workers, 3/3 trials; and 420 cuOpt solves on a single persistent worker in one process. So the corrupting condition needs a large, long-lived, allocation-diverse process. What does reproduce, intermittently but reliably at ~2 crashes in 3 runs:

• one CPython process;
• ~100 in-process cuOpt solves, mixed LP and MILP, all on one long-lived worker thread (i.e. with the original report's workaround applied), models released after each;
• then several minutes of heavy unrelated allocation churn — other native solvers (SCIP, HiGHS, CPLEX), numpy/xarray array work, CPython cyclic GC;
• crash arrives 7–12 minutes in, always after all cuOpt work has finished, always in unrelated native code, at a different site each time.

Our full --run-gpu suite is the reproducer; happy to share the exact invocation, any of the gdb logs, or to run instrumented builds or patched wheels on this machine if that would help. If it is useful we can also try to shrink it further with a hint about which libomp allocation paths to stress.

Relation to existing issues

As far as we can tell this is new on the tracker — we searched for heap corruption, _int_malloc, malloc.c, fd_nextsize, glibc, malloc corruption, double free and SIGSEGV malloc and found no existing report of this manifestation. Closest neighbours, all distinct:

• [BUG] MIP SIGSEGV - Multiple OpenMP runtime conflict #1219 (open, "MIP SIGSEGV — Multiple OpenMP runtime conflict") — a SIGSEGV inside libgomp during post-solve teardown of one large MIP. Here the crash is in glibc's allocator or in an unrelated library, arbitrarily long after teardown completed, and needs no large model.
• Fix libomp ABI bug #1593 (merged, "Fix libomp ABI bug") — a GOMP_task compatibility-layer bug in the same bundled LLVM libomp, i.e. prior art that this runtime's GCC-compat paths have had memory-safety defects. Different symptom (std::bad_array_new_length via an unconstructed firstprivate copy), but it is why we think the bundled libomp is the right place to look.
• [BUG] MIP solver thread count degrades on 2nd+ solve due to omp_get_max_threads() contamination #900 (closed) — OpenMP state surviving across sequential solves, but its symptom is degraded parallelism, not corruption.

The practical impact worth flagging: for the kmp_alloc assertion in this issue there is a workaround (never let a solver-entering thread exit). For this one there is no caller-side workaround we could find — process isolation is the only mitigation, since by the time the fault happens cuOpt is no longer on any stack.