Skip to content

0.0.19: Monty() lazily imports pydantic_monty._binary, hanging Temporal workflows #557

Description

@dsfaccini

David's AICA here:

Summary

In 0.0.19, constructing a Monty() pool triggers import pydantic_monty._binary
lazily (to locate the monty worker binary). When Monty() is first constructed
inside a Temporal workflow, that import happens during workflow execution.
Temporal's workflow sandbox forbids importing a module that was not already loaded
at "initial workflow load" and raises:

Module pydantic_monty._binary was imported after initial workflow load.

Temporal turns that into a workflow-task failure, and it retries a failed workflow
task indefinitely, so the workflow never completes -- the caller hangs rather than
getting an error.

This is a 0.0.19 change in behavior relative to 0.0.18: the in-process interpreter
had no lazy _binary import, so running Monty inside a Temporal workflow completed.

Diagnosis

pydantic_monty._binary is the only module imported lazily at pool construction
(verified with a sys.modules diff):

  • import pydantic_monty loads pydantic_monty, pydantic_monty._monty, pydantic_monty.os_access
  • the first Monty() then adds pydantic_monty._binary

Pre-loading pydantic_monty._binary before the workflow starts makes Monty run
correctly inside the Temporal workflow sandbox (the subprocess spawn and blocking IPC
are fine there once the Python import is already resolved). So the lazy import is the
sole blocker.

Suggested fix

Import _binary eagerly at package import (e.g. in pydantic_monty/__init__), so it
is always loaded with the package and never trips Temporal's post-load import guard.
Downstream code can work around it today with a top-level import pydantic_monty._binary,
but making the package Temporal-sandbox-safe by default avoids the footgun.

Minimal reproduction

Standalone -- no pydantic-ai / harness, only pydantic-monty, pydantic-monty-runtime,
temporalio. Constructs Monty() inside a Temporal workflow. With the bug present it
prints the import-after-load warning and a KeyError: 'pydantic_monty._binary', fails
the workflow task repeatedly, and (because Temporal retries indefinitely) never
completes -- the 60s asyncio.wait_for guard raises TimeoutError.

# /// script
# requires-python = ">=3.13,<3.14"
# dependencies = [
#     "pydantic-monty==0.0.19b2",
#     "pydantic-monty-runtime==0.0.19b2",
#     "temporalio>=1.9",
# ]
# ///
"""Minimal reproduction: constructing a `pydantic_monty.Monty` pool inside a
Temporal workflow hangs the workflow (lazy `pydantic_monty._binary` import trips
Temporal's sandbox no-post-load-import rule). Run with: uv run mre_monty_temporal.py
"""

from __future__ import annotations

import asyncio
import uuid

from pydantic_monty import Monty
from temporalio import workflow
from temporalio.testing import WorkflowEnvironment
from temporalio.worker import Worker

TASK_QUEUE = 'monty-temporal-mre'


@workflow.defn
class MontyWorkflow:
    @workflow.run
    async def run(self) -> str:
        # Constructing the pool triggers `import pydantic_monty._binary`, which the
        # Temporal sandbox rejects because it was not loaded at initial workflow load.
        with Monty() as pool:
            with pool.checkout() as session:
                session.feed_start('1 + 1')
        return 'ok'


async def main() -> None:
    async with await WorkflowEnvironment.start_local(port=7245) as env:
        async with Worker(
            env.client,
            task_queue=TASK_QUEUE,
            workflows=[MontyWorkflow],
        ):
            print('executing workflow (hangs here if the bug is present)...')
            result = await asyncio.wait_for(
                env.client.execute_workflow(
                    MontyWorkflow.run,
                    id=f'monty-mre-{uuid.uuid4()}',
                    task_queue=TASK_QUEUE,
                ),
                timeout=60,
            )
            print('workflow result:', result)


if __name__ == '__main__':
    asyncio.run(main())

Environment

  • Python 3.13 (Linux)
  • pydantic-monty==0.0.19b2, pydantic-monty-runtime==0.0.19b2
  • temporalio (latest); Temporal CLI 1.7.3 / Server 1.31.2

Context

Surfaced while migrating pydantic-ai-harness's code_mode capability to the 0.0.19
beta: its Temporal integration test hung in CI. We've worked around it downstream with
an eager import pydantic_monty._binary, but filing here since an eager import in
pydantic_monty would fix it for every Temporal user.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions