Skip to content

daemon: fix 100% CPU spin on long-running kernel executions#13

Merged
anish749 merged 1 commit into
mainfrom
fix/daemon-cpu-spin
Mar 30, 2026
Merged

daemon: fix 100% CPU spin on long-running kernel executions#13
anish749 merged 1 commit into
mainfrom
fix/daemon-cpu-spin

Conversation

@anish749

Copy link
Copy Markdown
Owner

Problem

The daemon process burns 100% CPU whenever a kernel execution takes longer than 10 seconds. This was traced to a busy-wait loop inside jupyter_kernel_client's execute_interactive method.

The library sets a deadline based on its REQUEST_TIMEOUT (default 10s). After the deadline passes, the message-polling loop degrades:

while True:
    timeout = max(0, deadline - time.monotonic())  # → 0
    self._message_received.wait(timeout=0)          # returns immediately
    msg = self.iopub_channel.get_msg(timeout=0)     # queue.Empty → continue
    # tight loop, burning CPU

A secondary issue: after a successful execution, _exec_with_disconnect cancels the disconnect watcher (reader.read(1)) but never awaits the cancelled task. This leaves the StreamReader in a corrupted state, causing readuntil() called while another coroutine is already waiting for incoming data errors on every subsequent request for that client connection.

Solution

  1. Pass timeout=None to kernel.execute() — disables the library's internal execution timeout entirely. The daemon already handles cancellation at a higher level via _exec_with_disconnect, which races execution against a client socket read and interrupts the kernel on disconnect. The library-level timeout is redundant and harmful.

  2. Await the cancelled disconnect watcher — after disconnect.cancel(), we now await disconnect (catching CancelledError) so the cancellation propagates through the reader's internals and properly releases its _wait_for_data state.

Context

The CLI's --timeout flag is a separate concern — it controls how long the client waits for a daemon response on the socket, not how long the kernel is allowed to run.

Disable the jupyter_kernel_client library-level execution timeout by
passing timeout=None to kernel.execute(). The library defaults to a 10s
timeout (REQUEST_TIMEOUT), after which its execute_interactive loop
degrades into a busy-wait: Event.wait(timeout=0) returns immediately,
and the while-loop spins burning a full CPU core.

The daemon does not need this timeout — client disconnection is already
handled by _exec_with_disconnect racing execution against a socket read.

Also fix a StreamReader corruption bug in _exec_with_disconnect: after
cancelling the disconnect watcher (reader.read(1)), the cancelled task
must be awaited so the CancelledError propagates through the reader
internals and releases its _wait_for_data state. Without this, the next
reader.readline() raises "readuntil() called while another coroutine is
already waiting for incoming data".
@anish749 anish749 merged commit 2d2cd5d into main Mar 30, 2026
1 check passed
@anish749 anish749 deleted the fix/daemon-cpu-spin branch March 30, 2026 10:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant