Cancelling MontyRepl.feed_run_async permanently poisons the session (panic at intern.rs:916)

[fugu-ai]

Summary

If an in-flight feed_run_async is cancelled (e.g. via asyncio.wait_for timeout), the REPL session can be left permanently broken: the next feed_run_async panics on a tokio worker at crates/monty/src/intern.rs:916 (index out of bounds: the len is 2 but the index is 2) and raises RuntimeError: Async REPL transition was cancelled before completion — and every subsequent feed on that instance fails the same way.

The failure is deterministic, not a race: it happens whenever the cancelled snippet stored a new string literal of 2+ chars into state that survives the cancellation. Values that (presumably) aren't interned — 1-char strings, ints — recover fine, which is what makes the bug easy to miss in stress tests.

Repro

# uv run --with pydantic-monty==0.0.18 python repro.py
import asyncio
from pydantic_monty import MontyRepl

async def main():
    repl = MontyRepl()
    await repl.feed_run_async("items = []")

    async def slow():
        await asyncio.sleep(5)

    try:
        await asyncio.wait_for(
            repl.feed_run_async(
                "items.append('side-effect')\nawait slow()",
                external_functions={"slow": slow},
            ),
            timeout=0.3,
        )
    except asyncio.TimeoutError:
        pass

    print("recovered:", await repl.feed_run_async("items"))  # panics here

asyncio.run(main())

Output:

thread 'tokio-rt-worker' panicked at crates/monty/src/intern.rs:916:9:
index out of bounds: the len is 2 but the index is 2
...
RuntimeError: Async REPL transition was cancelled before completion

What determines panic vs. recovery

Same scenario, varying only the appended literal (fresh process each trial, 3–5 trials per row, 100% consistent):

cancelled run stored	next feed
items.append('x')	recovers, partial mutation visible
items.append(12345)	recovers, partial mutation visible
items.append('ab')	panic
items.append('abcd')	panic
items.append('side-effect')	panic

The 1-char/2-char boundary and the intern.rs location suggest: cancellation rolls back the intern table's additions from the aborted transition, but heap objects mutated before the cancel still reference the newly-interned entry, so the next heap access indexes past the rolled-back table (len 2, index 2 — exactly one new interned string on top of the two baseline entries). That's a hypothesis from black-box behavior; we haven't read the interning code.

Environment

• pydantic-monty 0.0.18 (current latest), macOS 15 arm64
• Reproduced on CPython 3.12.8 and 3.13.14
• 25/25 failures in fresh processes with the repro above; also reproduces mid-session in long-lived processes

Context and workaround

We hit this building an agent code-execution runtime (toolplane) that enforces a per-run timeout with asyncio.wait_for around feed_run_async. Sessions whose timed-out snippet had touched any realistic string data were bricked afterwards.

Workaround that holds up: dump() before each feed; on cancellation, discard the instance entirely and MontyRepl.load(snapshot) a fresh one. (That also gives run-level atomicity, since completed statements of a cancelled run otherwise persist — see the recovery rows above — which may or may not be intended semantics, but the panic rows clearly aren't.)

Happy to test a fix branch.

[samuelcolvin]

You should try on main a lot has changed.

[fugu-ai]

Retested on main @ 5de8e00, built from source (macOS arm64, CPython 3.12), ported to the new AsyncMonty pool API. Results:

The panic is gone — no more intern.rs crash in any variant we could produce. Nice.

But cancellation still permanently wedges the session, now uniformly (the string-interning discriminator from the original report no longer matters — short strings and ints wedge it too):

async with AsyncMonty() as monty:
    async with monty.checkout() as s:
        await s.feed_run("items = []")
        try:
            await asyncio.wait_for(
                s.feed_run("items.append('x')\nawait slow()",
                           external_lookup={"slow": slow}),
                timeout=0.3)
        except asyncio.TimeoutError:
            pass
        await s.feed_run("items")
        # RuntimeError: monty worker protocol error:
        #   feed called while a suspension is awaiting an answer

Every subsequent feed_run on that session fails the same way, and same-session recovery is blocked by design (load / load_snapshot is only valid on a fresh session).

This is a real improvement in failure quality — a clean, catchable RuntimeError instead of a native panic with heap corruption — and the workaround is cleaner on the new API too (verified): await s.dump() before the risky feed, discard the checkout, monty.checkout() + load(snap) restores the pre-run state (138-byte snapshot in our probe).

Open question for the API: is discard-and-restore the intended pattern for host-side timeouts, or is an explicit way to abandon an in-flight suspension (so the same session survives a cancelled feed) on the roadmap? Host-enforced per-feed timeouts seem like a common need for agent runtimes, and today the first timeout costs you the worker. Happy to test any branch.

[fugu-ai]

Status on 0.0.19b2 (macOS arm64): the intern.rs panic is gone. Cancelling a feed_run mid-suspension now wedges the session deterministically — every later feed raises monty worker protocol error: feed called while a suspension is awaiting an answer, and same-session load() is rejected (only valid on a fresh session). Our recovery pattern works cleanly on the pool API: dump() before the run, discard the wedged checkout, load() the snapshot into a fresh checkout.

One sharp edge in that recovery is new: if the cancelled feed was CPU-busy rather than suspended, closing the wedged checkout hangs forever — filed separately as #551.

Related question while you're in this area: is it intended that protocol-level errors (e.g. No pending async tasks but ResolveFutures requested from awaiting a future persisted by an earlier feed) now finish the checkout? On 0.0.18 the REPL survived them with prior state intact; on the beta the next feed raises this checkout has already been finished. We recover via the same snapshot-into-fresh-checkout dance, but if it's intentional it might deserve a doc note since it silently rolls back statements the failing feed completed.