Problem
The codegen emit/load tests exercise a round trip: emit generated TTNN Python (a dry run) then load and execute it via the embedded PythonModelRunner. The load phase requires the plugin to be built with TTXLA_ENABLE_EMITPY_EXECUTION=ON.
The CI manylinux wheel is built with TTXLA_ENABLE_EMITPY_EXECUTION=OFF on purpose (python_package/setup.py, the in_ci() branch — "avoid depending on the embedded-Python runner and its non-portable transitive ABI surface"). So on the CI wheel the load path returns kInternal (error code 13):
so_loaded_executable_in:149 ERR| EmitPy execution requested, but this build does not include PythonModelRunner support
RuntimeError: Bad StatusOr access: INTERNAL: Error code: 13
The tests pass on local editable/dev builds (EmitPy defaults ON) but fail in CI (e.g. run 28452859580, PR #5160).
Current stopgap
The affected tests are skipped in CI via pytest.mark.skipif(os.environ.get("GITHUB_ACTIONS") == "true", ...):
tests/torch/codegen/test_codegen_emit_load.py
tests/torch/codegen/test_codegen_emit_load_multichip.py
This is a blunt gate — it keys off "am I in GitHub Actions" rather than "was the plugin built with EmitPy", so it would also skip on a hypothetical CI build that does include EmitPy, and it drops all CI coverage of the feature.
To investigate
- Can the manylinux wheel bundle
PythonModelRunner portably (resolve the non-portable transitive ABI surface) so the load path is testable in CI?
- Or expose the build's EmitPy capability to Python (e.g. a generated
pjrt_plugin_tt/build_config.py, or a plugin attribute) and skip precisely on capability instead of on CI detection.
- Or run these tests in a dedicated job/pipeline that builds with
TTXLA_ENABLE_EMITPY_EXECUTION=ON.
Refs
python_package/setup.py — in_ci() → TTXLA_ENABLE_EMITPY_EXECUTION=OFF
pjrt_implementation/src/api/so_loaded_executable_instance.cc — the #if TTXLA_ENABLE_EMITPY_EXECUTION guarded execution path
Problem
The codegen emit/load tests exercise a round trip: emit generated TTNN Python (a dry run) then load and execute it via the embedded
PythonModelRunner. The load phase requires the plugin to be built withTTXLA_ENABLE_EMITPY_EXECUTION=ON.The CI manylinux wheel is built with
TTXLA_ENABLE_EMITPY_EXECUTION=OFFon purpose (python_package/setup.py, thein_ci()branch — "avoid depending on the embedded-Python runner and its non-portable transitive ABI surface"). So on the CI wheel the load path returnskInternal(error code 13):The tests pass on local editable/dev builds (EmitPy defaults ON) but fail in CI (e.g. run 28452859580, PR #5160).
Current stopgap
The affected tests are skipped in CI via
pytest.mark.skipif(os.environ.get("GITHUB_ACTIONS") == "true", ...):tests/torch/codegen/test_codegen_emit_load.pytests/torch/codegen/test_codegen_emit_load_multichip.pyThis is a blunt gate — it keys off "am I in GitHub Actions" rather than "was the plugin built with EmitPy", so it would also skip on a hypothetical CI build that does include EmitPy, and it drops all CI coverage of the feature.
To investigate
PythonModelRunnerportably (resolve the non-portable transitive ABI surface) so the load path is testable in CI?pjrt_plugin_tt/build_config.py, or a plugin attribute) and skip precisely on capability instead of on CI detection.TTXLA_ENABLE_EMITPY_EXECUTION=ON.Refs
python_package/setup.py—in_ci()→TTXLA_ENABLE_EMITPY_EXECUTION=OFFpjrt_implementation/src/api/so_loaded_executable_instance.cc— the#if TTXLA_ENABLE_EMITPY_EXECUTIONguarded execution path