[FEATURE] Add a Toxiproxy-based fault-injection integration test for ungraceful TCP disconnects (zombie registrations / swallowed RPCs)

[oberstet]

Type: testing / CI-CD (network resilience)
Priority: medium — deferred (post-26.6.1)
Related: #2187 (silently swallowed RPC calls), #2130 (registration leak when
session close throws), #2153 (caller disconnect / callee crash)

Summary

Add an end-to-end fault-injection integration test that reproduces the
real-world failure mode behind #2187 — ungracefully dropped / half-open TCP
connections leaving "zombie" callee registrations that silently swallow
round-robin RPC calls — using Toxiproxy
to inject network faults between WAMP clients and the Crossbar.io router.

This complements (does not replace) the deterministic Dealer unit test
added under #2187. The unit test proves the specific dealer defect is fixed;
this integration test proves the real customer scenario reproduces and that
Crossbar.io's heartbeat / auto-ping machinery actually detects and cleans up dead
callees end-to-end, over a real (faulted) network path.

Motivation — why this matters for users/customers

1. Production networks are unreliable. Crossbar.io deployments routinely span
   corporate proxies, cloud regions, NAT, and lab-to-cloud links (the [BUG] silently swallowed RPC calls #2187
   reporter runs a globally distributed device farm, lab ↔ Azure). Ungraceful TCP
   teardown (no FIN/RST — blackhole, middlebox idle-eviction, power loss, cable
   pull) is common in the field, not an edge case.
2. Silent data loss is the worst failure mode. [BUG] silently swallowed RPC calls #2187 manifested as RPC calls
   that simply never complete, with the offending registrations invisible to
   wamp.registration.list — extremely hard to diagnose in production. A test
   that exercises this path guards an entire class of zombie-registration /
   swallowed-call bugs ([BUG] silently swallowed RPC calls #2187, Registration leak when session closing throws exception #2130, Test/caller disconnect callee crash #2153) at the network level, where unit
   tests can't reach.
3. Validates the heartbeat contract. Crossbar's WebSocket auto-ping/auto-
   timeout (and reconnect timing interplay with the router's heartbeat) is the
   mechanism that's supposed to detect and reap half-open connections. Today that
   contract is only reasoned about, not tested under real fault injection. This
   test turns "should detect dead peers" into an enforced, regression-guarded
   guarantee.
4. Customer credibility / marketing. A reproducible, automated fault-
   injection harness demonstrating production-grade resilience under adverse
   networks is a strong signal for high-profile users (e.g. the [BUG] silently swallowed RPC calls #2187 reporter's
   device-farm platform) and for evaluating Crossbar.io in demanding deployments.
5. Reusable capability. A Toxiproxy harness unlocks a whole family of future
   network-resilience tests (latency, bandwidth caps, jitter, partial failures,
   slow-loris, reconnect storms) — not just this one scenario.

Proposed approach (Phase 1)

A Docker Compose-based integration scenario (see the discussion in #2187):

• Topology: crossbar + toxiproxy + a test driver (Autobahn|Python WAMP
  clients), composed via Docker Compose. WAMP callee sessions connect to the
  router through Toxiproxy so faults can be injected per connection.
• Setup: a callee agent opens N sessions and registers the same procedure on
  each with invoke: "roundrobin"; a caller invokes the procedure repeatedly.
• Fault injection: via the Toxiproxy API, blackhole one (or K of N) callee
  connections ungracefully (e.g. timeout toxic / drop without FIN-RST) to
  simulate a half-open TCP path.
• Assertions:
  • while faulted, round-robin invocations must not be silently swallowed
    indefinitely (no permanent K/N loss);
  • Crossbar's auto-ping/auto-timeout must detect the dead callee within the
    configured heartbeat window and clean up its session + registration
    (round-robin group becomes consistent again);
  • re-registration (incl. non-roundrobin policies) must not fail with
    wamp.error.procedure_exists_with_different_invocation_policy;
  • registrations visible via the meta API match the actual live callees.
• Knobs: parametrize auto_ping_interval / auto_ping_timeout (router and
  client) and reconnect delay, so the test pins down the timing contract.

CI considerations

• Network-timing-dependent → can be flaky. Run it as a dedicated / nightly
  job (or a clearly separated, generously-timed integration stage), not as a
  fast-path PR gate, to avoid destabilizing PR CI. Use deterministic toxics
  (hard blackhole) over probabilistic ones where possible, and assert on
  eventual cleanup within a bounded, configurable timeout.
• New just recipe (e.g. test-integration-toxiproxy) + Docker Compose assets
  under the integration-test tree; reuse existing Docker/CI patterns where
  possible.

Non-goals

• The unit-level Dealer.detach() fix and its deterministic regression test —
  that is [BUG] silently swallowed RPC calls #2187 and ships in 26.6.1. This issue is the complementary end-to-end
  network reproduction and is intentionally deferred to a later release.

---

_Note: This work was completed with AI assistance (Claude Code). Initiated and reviewed by me. _

Checklist

• I have searched existing issues to avoid duplicates
• I have described the problem clearly
• I have provided use cases
• I have considered alternatives
• I have assessed impact and breaking changes