Skip to content

Commit b8766d8

Browse files
committed
Use the "spawn" start method instead of forkserver
Python 3.14 defaults to the "forkserver" multiprocessing start method, instead of the previous "fork" default. This new default causes all sorts of problems for us. The hardest to fix one, which made me throw in the towel on trying to get things working with "forkserver", is that pytest's `pytester` fixture defaults to running the new pytest session in the current process, rather than from a subprocess, and we're using `capfd` to capture the stderr. With the "fork" method and the "spawn" method, the new process inherits whatever the parent process's stderr (fd 2) is pointed to at the time when the new process is started, but with "forkserver" the server gets created once and then reused, and all future children have their stderr pointed wherever the first child process's stderr was pointed to. There's other issues too. We leak semaphores because the monitoring process winds up creating an unnecessary `Event` and `Queue` that it ignores in favor of the ones passed to its entry point. That could be fixed by creating those lazily instead of at import time, but that leads to pickling failures because the `multiprocessing.connection` module gets created lazily, and `pytester` unloads modules that were loaded by the tests, which leads to two different copies of that module getting loaded, with the old module still referenced by some things. We could work around all of this in our test suite, but using our plugin along with the `pytester` fixture is a reasonable use case that end users might want, if they're trying to use `pytest-pystack` to test their own pytest plugin. The most reasonable solution is for us to detect when "forkserver" would be used, and to use "spawn" instead in those cases.
1 parent 6306d63 commit b8766d8

3 files changed

Lines changed: 13 additions & 5 deletions

File tree

src/pytest_pystack/_debug_detect.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
import inspect # used for introspection of module name
2-
import multiprocessing
32
import sys
43

5-
debug_detected = multiprocessing.Event()
4+
from ._multiprocessing_context import MP_CTX
5+
6+
debug_detected = MP_CTX.Event()
67
# bdb covers pdb, ipdb, and possibly others
78
# pydevd covers PyCharm, VSCode, and possibly others
89
KNOWN_DEBUGGING_MODULES = {"pydevd", "bdb", "pydevd_frame_evaluator"}

src/pytest_pystack/_monitor_process.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
import multiprocessing
21
import os
32
import shlex
43
import subprocess
@@ -7,14 +6,15 @@
76

87
from ._config import PystackConfig
98
from ._debug_detect import debug_detected
9+
from ._multiprocessing_context import MP_CTX
1010

11-
_queue = multiprocessing.Queue()
11+
_queue = MP_CTX.Queue()
1212
_process = None
1313

1414

1515
def start(config):
1616
global _process
17-
_process = multiprocessing.Process(
17+
_process = MP_CTX.Process(
1818
target=_run_monitor,
1919
args=(
2020
config,
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
import multiprocessing
2+
3+
# Ensure the "forkserver" spawn method is never used, because we have
4+
# known incompatibilities with it, while both "fork" and "spawn" work.
5+
MP_CTX = multiprocessing.get_context(
6+
"fork" if multiprocessing.get_start_method() == "fork" else "spawn"
7+
)

0 commit comments

Comments
 (0)