daemon: fix 100% CPU spin on long-running kernel executions#13
Merged
Conversation
Disable the jupyter_kernel_client library-level execution timeout by passing timeout=None to kernel.execute(). The library defaults to a 10s timeout (REQUEST_TIMEOUT), after which its execute_interactive loop degrades into a busy-wait: Event.wait(timeout=0) returns immediately, and the while-loop spins burning a full CPU core. The daemon does not need this timeout — client disconnection is already handled by _exec_with_disconnect racing execution against a socket read. Also fix a StreamReader corruption bug in _exec_with_disconnect: after cancelling the disconnect watcher (reader.read(1)), the cancelled task must be awaited so the CancelledError propagates through the reader internals and releases its _wait_for_data state. Without this, the next reader.readline() raises "readuntil() called while another coroutine is already waiting for incoming data".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The daemon process burns 100% CPU whenever a kernel execution takes longer than 10 seconds. This was traced to a busy-wait loop inside
jupyter_kernel_client'sexecute_interactivemethod.The library sets a deadline based on its
REQUEST_TIMEOUT(default 10s). After the deadline passes, the message-polling loop degrades:A secondary issue: after a successful execution,
_exec_with_disconnectcancels the disconnect watcher (reader.read(1)) but never awaits the cancelled task. This leaves theStreamReaderin a corrupted state, causingreaduntil() called while another coroutine is already waiting for incoming dataerrors on every subsequent request for that client connection.Solution
Pass
timeout=Nonetokernel.execute()— disables the library's internal execution timeout entirely. The daemon already handles cancellation at a higher level via_exec_with_disconnect, which races execution against a client socket read and interrupts the kernel on disconnect. The library-level timeout is redundant and harmful.Await the cancelled disconnect watcher — after
disconnect.cancel(), we nowawait disconnect(catchingCancelledError) so the cancellation propagates through the reader's internals and properly releases its_wait_for_datastate.Context
The CLI's
--timeoutflag is a separate concern — it controls how long the client waits for a daemon response on the socket, not how long the kernel is allowed to run.