Skip to content

Test/caller disconnect callee crash - #2153

Merged
oberstet merged 4 commits into
crossbario:masterfrom
RecordEvolution:test/caller-disconnect-callee-crash
Jan 4, 2026
Merged

Test/caller disconnect callee crash#2153
oberstet merged 4 commits into
crossbario:masterfrom
RecordEvolution:test/caller-disconnect-callee-crash

Conversation

@markope

@markope markope commented Jan 3, 2026

Copy link
Copy Markdown
Contributor

a rundown on what the issue is and how it impacts the callee:

Exception Flow: Caller Disconnect Causes Callee Session Crash

When a caller disconnects during an in-flight RPC, the callee's session can be terminated due to unhandled exception propagation:

The Problem

  1. Caller disconnects during an in-flight RPC call
  2. Callee completes work and sends YIELD message with the result
  3. Dealer receives YIELD but the invocation tracking data is stale (points to disconnected caller)
  4. processYield() in dealer.py tries to send result to caller._transport which is None
  5. Exception is raised → propagates up to router.process()
  6. router.process() re-raises ProtocolError (see router.py#L355-356)
  7. Exception bubbles to Autobahn's WampWebSocketProtocol.onMessage()
  8. Autobahn catches ProtocolError and calls _bailout() with CLOSE_STATUS_CODE_PROTOCOL_ERROR
  9. WebSocket connection is closedCallee's session is terminated

Code Path

Callee sends YIELD
  → RouterSession.onMessage()
    → router.process()
      → dealer.processYield()
        → Exception (stale invocation / None transport)
      → ProtocolError re-raised
    → Exception propagates (no catch)
  → WampWebSocketProtocol.onMessage() catches ProtocolError
    → _bailout() → _fail_connection()
      → WebSocket closed → Callee session LOST

Impact

Any ProtocolError raised in the dealer causes the entire session's WebSocket connection to be closed, which is catastrophic for the callee who did nothing wrong.

The Fix

Clean up invocation tracking when the caller detaches, so that when the callee later calls processYield():

  • The invocation is already gone from _invocations
  • processYield() simply logs a debug message and returns None
  • No exception is raised
  • Callee's session remains intact

The Test

The test creates the scenario sketched above and expects the dealer.py to not raise an error in step 4 (processYield).
Currently without the fix the test does not pass.

@markope

markope commented Jan 3, 2026

Copy link
Copy Markdown
Contributor Author

tests #2133

Failed test visible here

When a caller disconnects while an RPC is in-flight, the router must
clean up invocation tracking regardless of whether the callee supports
call canceling. Previously, the cleanup was skipped if the callee
didn't support call_canceling, leaving stale data structures that
caused errors when the callee tried to return a result.

The fix:
- Always clean up invocation tracking (timeout, data structures)
- Only send INTERRUPT message if callee supports call_canceling
- Log appropriately in both cases

Fixes: crossbario#2133
@markope

markope commented Jan 3, 2026

Copy link
Copy Markdown
Contributor Author

all tests pass now see here

@markope

markope commented Jan 3, 2026

Copy link
Copy Markdown
Contributor Author

@oberstet I hope I followed your required procedure here. I motivated the issue more clearly in the initial PR comment above. The comments in the new test case give some additional details about the nature of the test and why the fix is a fix.

@oberstet

oberstet commented Jan 4, 2026

Copy link
Copy Markdown
Contributor

fantastic! this is a showcase example for the workflow!

it's perfect - both for me reviewing, as well as (IMO) for users.

thanks also in particular for the detailed comments on the new test test_caller_detach_cleans_up_invocation_tracking_without_cancel_support()!

ah, and the remaining failed code quality check was completely unrelated to this PR.

merging.

@oberstet
oberstet merged commit 5e591ab into crossbario:master Jan 4, 2026
17 of 18 checks passed
@oberstet

oberstet commented Jan 4, 2026

Copy link
Copy Markdown
Contributor

love it. thanks so much for the fix! and for taking so much care following the workflow.

fwiw, now (together with fixed typo wrt check-typing), workflows are 100% green & happy:

image

the release.yml - correctly - posted a new nightly/development github release: https://github.com/crossbario/crossbar/releases/tag/master-202601040913

image

@oberstet

oberstet commented Jan 4, 2026

Copy link
Copy Markdown
Contributor

what is still missing here is: release-post-comment.yml - which posts a github discussions post in CI-CD (https://github.com/crossbario/crossbar/discussions/categories/ci-cd) => filed as #2154

mumbacloud2025 pushed a commit to mumbacloud2025/crossbar that referenced this pull request Mar 6, 2026
When a caller disconnects during an in-flight RPC, invocation cleanup was
skipped if the callee did not support call_canceling (due to an early
continue). This left stale entries in _invocations and _callee_to_invocations,
causing errors when the callee later tried to return a result.

Fix: move cleanup before the call_canceling check so it always runs.
Only the INTERRUPT message is conditional on call_canceling support.

Port of upstream crossbar PR crossbario#2153 (fixes issue crossbario#2133).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
mumbacloud2025 pushed a commit to mumbacloud2025/crossbar that referenced this pull request Mar 9, 2026
Port of upstream PR crossbario#2153: when a caller disconnects during an in-flight
RPC, invocation cleanup was skipped if the callee did not support
call_canceling (due to an early continue). This left stale entries in
_invocations and _callee_to_invocations, causing errors when the callee
later tried to return a result.

Fix: move cleanup before the call_canceling check so it always runs.
Only the INTERRUPT message is conditional on call_canceling support.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants