David's AICA here:
Summary
In 0.0.19, constructing a Monty() pool triggers import pydantic_monty._binary
lazily (to locate the monty worker binary). When Monty() is first constructed
inside a Temporal workflow, that import happens during workflow execution.
Temporal's workflow sandbox forbids importing a module that was not already loaded
at "initial workflow load" and raises:
Module pydantic_monty._binary was imported after initial workflow load.
Temporal turns that into a workflow-task failure, and it retries a failed workflow
task indefinitely, so the workflow never completes -- the caller hangs rather than
getting an error.
This is a 0.0.19 change in behavior relative to 0.0.18: the in-process interpreter
had no lazy _binary import, so running Monty inside a Temporal workflow completed.
Diagnosis
pydantic_monty._binary is the only module imported lazily at pool construction
(verified with a sys.modules diff):
import pydantic_monty loads pydantic_monty, pydantic_monty._monty, pydantic_monty.os_access
- the first
Monty() then adds pydantic_monty._binary
Pre-loading pydantic_monty._binary before the workflow starts makes Monty run
correctly inside the Temporal workflow sandbox (the subprocess spawn and blocking IPC
are fine there once the Python import is already resolved). So the lazy import is the
sole blocker.
Suggested fix
Import _binary eagerly at package import (e.g. in pydantic_monty/__init__), so it
is always loaded with the package and never trips Temporal's post-load import guard.
Downstream code can work around it today with a top-level import pydantic_monty._binary,
but making the package Temporal-sandbox-safe by default avoids the footgun.
Minimal reproduction
Standalone -- no pydantic-ai / harness, only pydantic-monty, pydantic-monty-runtime,
temporalio. Constructs Monty() inside a Temporal workflow. With the bug present it
prints the import-after-load warning and a KeyError: 'pydantic_monty._binary', fails
the workflow task repeatedly, and (because Temporal retries indefinitely) never
completes -- the 60s asyncio.wait_for guard raises TimeoutError.
# /// script
# requires-python = ">=3.13,<3.14"
# dependencies = [
# "pydantic-monty==0.0.19b2",
# "pydantic-monty-runtime==0.0.19b2",
# "temporalio>=1.9",
# ]
# ///
"""Minimal reproduction: constructing a `pydantic_monty.Monty` pool inside a
Temporal workflow hangs the workflow (lazy `pydantic_monty._binary` import trips
Temporal's sandbox no-post-load-import rule). Run with: uv run mre_monty_temporal.py
"""
from __future__ import annotations
import asyncio
import uuid
from pydantic_monty import Monty
from temporalio import workflow
from temporalio.testing import WorkflowEnvironment
from temporalio.worker import Worker
TASK_QUEUE = 'monty-temporal-mre'
@workflow.defn
class MontyWorkflow:
@workflow.run
async def run(self) -> str:
# Constructing the pool triggers `import pydantic_monty._binary`, which the
# Temporal sandbox rejects because it was not loaded at initial workflow load.
with Monty() as pool:
with pool.checkout() as session:
session.feed_start('1 + 1')
return 'ok'
async def main() -> None:
async with await WorkflowEnvironment.start_local(port=7245) as env:
async with Worker(
env.client,
task_queue=TASK_QUEUE,
workflows=[MontyWorkflow],
):
print('executing workflow (hangs here if the bug is present)...')
result = await asyncio.wait_for(
env.client.execute_workflow(
MontyWorkflow.run,
id=f'monty-mre-{uuid.uuid4()}',
task_queue=TASK_QUEUE,
),
timeout=60,
)
print('workflow result:', result)
if __name__ == '__main__':
asyncio.run(main())
Environment
- Python 3.13 (Linux)
- pydantic-monty==0.0.19b2, pydantic-monty-runtime==0.0.19b2
- temporalio (latest); Temporal CLI 1.7.3 / Server 1.31.2
Context
Surfaced while migrating pydantic-ai-harness's code_mode capability to the 0.0.19
beta: its Temporal integration test hung in CI. We've worked around it downstream with
an eager import pydantic_monty._binary, but filing here since an eager import in
pydantic_monty would fix it for every Temporal user.
David's AICA here:
Summary
In 0.0.19, constructing a
Monty()pool triggersimport pydantic_monty._binarylazily (to locate the
montyworker binary). WhenMonty()is first constructedinside a Temporal workflow, that import happens during workflow execution.
Temporal's workflow sandbox forbids importing a module that was not already loaded
at "initial workflow load" and raises:
Temporal turns that into a workflow-task failure, and it retries a failed workflow
task indefinitely, so the workflow never completes -- the caller hangs rather than
getting an error.
This is a 0.0.19 change in behavior relative to 0.0.18: the in-process interpreter
had no lazy
_binaryimport, so running Monty inside a Temporal workflow completed.Diagnosis
pydantic_monty._binaryis the only module imported lazily at pool construction(verified with a
sys.modulesdiff):import pydantic_montyloadspydantic_monty,pydantic_monty._monty,pydantic_monty.os_accessMonty()then addspydantic_monty._binaryPre-loading
pydantic_monty._binarybefore the workflow starts makes Monty runcorrectly inside the Temporal workflow sandbox (the subprocess spawn and blocking IPC
are fine there once the Python import is already resolved). So the lazy import is the
sole blocker.
Suggested fix
Import
_binaryeagerly at package import (e.g. inpydantic_monty/__init__), so itis always loaded with the package and never trips Temporal's post-load import guard.
Downstream code can work around it today with a top-level
import pydantic_monty._binary,but making the package Temporal-sandbox-safe by default avoids the footgun.
Minimal reproduction
Standalone -- no pydantic-ai / harness, only
pydantic-monty,pydantic-monty-runtime,temporalio. ConstructsMonty()inside a Temporal workflow. With the bug present itprints the import-after-load warning and a
KeyError: 'pydantic_monty._binary', failsthe workflow task repeatedly, and (because Temporal retries indefinitely) never
completes -- the 60s
asyncio.wait_forguard raisesTimeoutError.Environment
Context
Surfaced while migrating
pydantic-ai-harness'scode_modecapability to the 0.0.19beta: its Temporal integration test hung in CI. We've worked around it downstream with
an eager
import pydantic_monty._binary, but filing here since an eager import inpydantic_montywould fix it for every Temporal user.