Query requests are silently dropped when using shared memory when receiver's RX thread stalls

[zygfrydw]

Describe the bug

When the transport-level shared-memory optimization (transport/shared_memory/transport_optimization) is enabled, query requests are auto-promoted into watchdog-protected SHM buffers. If the receiving side's RX thread is descheduled for longer than the watchdog TTL (~100–200 ms) between accepting consecutive frames, the SHM chunk is invalidated by the sender's watchdog before the receiver maps it and is able to confirm the SHM message.
The receiver then silently drops the query (return Ok(()) with only a tracing::debug!) and the querier never receives a reply or any information that the query has been dropped. Therefore, the client get() hangs full queries_default_timeout (default 600 s).

Mechanism (root cause)

1. The client issues session.get(...) request.
2. On the wire path, map_zmsg_to_partner auto-promotes the Request::Query payload into an SHM chunk.
3. Every allocated chunk is registered with the watchdog subsystem:
   • GLOBAL_CONFIRMATOR (period 50 ms, commons/zenoh-shm/src/watchdog/confirmator.rs) — the owner of a live ConfirmedDescriptor keeps "kicking" the chunk.
   • GLOBAL_VALIDATOR (period 100 ms, commons/zenoh-shm/src/watchdog/validator.rs) — if a chunk's bit hasn't been kicked since the last tick, it sets watchdog_invalidated = true.
4. After the sender's transport writes the frame to TCP and releases its chunk handle, only the receiver can keep the chunk alive, and only after it successfully calls read_shmbuf.
5. If the receiver's RX thread is stalled (CPU contention, etc.) and does not drain the socket within the watchdog window, the sender-side validator invalidates the chunk.
6. When the RX thread finally runs, map_zmsg_to_shmbuf → read_shmbuf → is_valid() fails (commons/zenoh-shm/src/lib.rs: !watchdog_invalidated && generation == info.generation) and returns bail!("Buffer is invalidated") (commons/zenoh-shm/src/reader.rs:79).
7. The transport RX swallows it:

 if let Some(shm_context) = &self.shm_context {
     if let Err(e) =
         crate::shm::map_zmsg_to_shmbuf(msg.as_mut(), &shm_context.shm_reader)
     {
         tracing::debug!("Error receiving SHM buffer: {e}");
         return Ok(());
     }
 }

8. The query request never reaches the queryable callback. The querier has a matched queryable, so it does not fire the "finished with 0 replies" drop path — it waits the full queries_default_timeout (default 600 s).

Impact

We hit this issue in production using rmw_zenoh, where it strongly affects ROS 2 services. RMW treats service calls as reliable and implements no retry logic for queries, so a single silently-dropped request (per the mechanism above) translates directly into a hung service call with no error surfaced to the application.

Our system comprises more than 100 ROS 2 nodes, with several grouped into composable containers. A composable container loads its components by issuing a sequence of ROS 2 service requests to load each component into the process. At system startup, all of these load requests are issued in a short burst while the machine is under heavy CPU contention — a thread storm competing for resources until the system stabilizes.
During this startup window, some of the component-load queries are silently dropped. Because rmw_zenoh does not retry and the get() simply hangs (up to queries_default_timeout), the affected containers are left partially initialized, with a non-deterministic set of components never loaded. The system comes up in a broken state, and the components that are missing vary from boot to boot.

Workaround

Disable shared memory transport optimization and use shared memory explicitly only for topics that tolerate lost messages.

To reproduce

I attach the source code for shm_query_client and shm_query_server and their respective configs.

1. Run service RUST_LOG=zenoh=debug ./shm_query_server configs/shm_query_server.json5 [client] loaded config=shm_query_server.json
2. Run client RUST_LOG=zenoh=debug ./shm_query_client shm_query_client.json5 8192 10000
   It will make the first request over TCP (lazy shared memory initialization), and it will wait for [ENTER]
3. Stop the service process to simulate CPU congestion with Ctrl+Z
4. Press Enter in shm_query_client to send the second query
5. Go back to the shm_query_server terminal and execute fg 1 to resume service.
6. When the RX thread in service resumes, it will print a debug log informing that the SHM segment got lost

2026-06-01T09:57:09.852157Z DEBUG  rx-0 ThreadId(07) zenoh_transport::unicast::universal::rx: Error receiving SHM buffer: Buffer is invalidated at /home/runner/.cargo/git/checkouts/zenoh-9c599d5ef3e0480e/81c6c93/commons/zenoh-shm/src/reader.rs:79.
2026-06-01T09:57:11.623704Z DEBUG  rx-1 ThreadId(08) zenoh_transport::unicast::universal::link: RX task failed: Read error on TCP link 127.0.0.1:7448 => 127.0.0.1:39906: early eof at /home/runner/.cargo/git/checkouts/zenoh-9c599d5ef3e0480e/81c6c93/io/zenoh-links/zenoh-link-tcp/src/unicast.rs:159.

7. The shm_query_client will be frozen until the query timeout elapses.

shm_query_client.cpp
shm_query_server.cpp
shm_query_client.json
shm_query_server.json

System info

• Zenoh version: zenohd v1.9.0, originally reproduced on 1.6.2 (b81e253)
• Platform / OS: Ubuntu 24.04.4 LTS (Noble Numbat)
• Kernel: Linux 6.17.0-29-generic, PREEMPT_DYNAMIC, x86_64
• CPU: Intel(R) Core(TM) i7-14700KF — 20 cores / 28 threads
• Architecture: x86_64
• Toolchain: gcc 13.3.0, cmake 3.28.3

[yellowhatter]

Hi @zygfrydw !
This is a serious issue. To be discussed.

[zygfrydw]

I'm afraid this issue carries significant weight: reliability is no longer guaranteed when shared memory is used, since a message can always be silently dropped.

I've been thinking about how this could be fixed, and I don't have a good solution yet - just a lot of ideas, all with drawbacks. Let me lay them out to kick off a brainstorm.

1. Retransmission. I think it's impossible to retransmit a message that was sent via SHM: the message is already gone from the publisher's queues and can't be sent again. If the publisher kept a copy, we'd hit the same cleanup and watchdog issues that SHM has today.

2. Increase RX thread priority and/or loosen the watchdog tolerance. The 100ms budget is very tight, so relaxing it would partially help. This is a tempting direction, and the timeout could be exposed as a configuration parameter to balance responsiveness to a crashed subscriber against tolerance for hiccups. However, it doesn't actually solve the problem - a dropped message can still happen with enough exposition, so the application must still be written as if any message could be dropped. And raising the priority of more and more threads erodes the benefit of higher priority altogether; during a boot-time thread storm, the issue can still occur.

3. Subscriber acknowledgment flags (a bit of a crazy idea). The SHM segment could carry a uint64 of flags, with each subscriber raising its bit to confirm it received the SHM segment handle. The problem is that this makes SHM validation on the publisher side very complex: the validator would have to check the link state of any subscriber that hasn't raised its flag and detect a crash based on link state — but that may be impossible over UDP. It also has hard limitations, such as a maximum number of subscribers, on top of the complex implementation.

4. Inform the publisher about the lost message. This is the best idea I have so far. The SHM handle carries enough information to tell the querier that its query was dropped and should be retried. However, it has several limitations:

   • It only works for queries. The publisher has no natural channel to receive this status - the TCP message was already delivered, so even in BLOCK mode the application thread has already returned from put().
   • It requires significant rework in ROS. There is currently no API for failed queries or retry logic. ROS 2 services provide no way to propagate a failed service call back to the application, so even an unavailable query expression will block that query indefinitely. I believe that should be fixed regardless, but today there's no easy way to notify the application.
   • The only quick-and-dirty fix would be at the RMW layer: implicitly retry when a service call fails due to an SHM issue.

5. Per-topic transport optimization. Provide an option to choose transport optimization per "topic," so that when the application configures a topic as reliable (BLOCK), SHM is not used. Currently this is only possible at the application level.

I have reported this issue in rmw_zehon for visibility and it seams to affect others too. Right now we're using a workaround - we disable transport_optimization so SHM is not used for queries, while the RMW layer uses SHM explicitly for pub/sub, accepting that some messages will be silently dropped.

[yellowhatter]

Historically, SHM was designed for low-latency operation cycle, which did not intend RX stalls, so this behavior leaked in and became a design flaw.

I think we need to make a solid solution to solve this.

Each SHM buffer carries watchdog reference, that is added and removed to/from Confirmator on RAII basis. Transport drops SHM buffer after sending it's packed descriptor to socket, removing watchdog descriptor from Confirmator. Mechanically, to achieve link-safety for watchdog, we need to drop it's descriptor only after link partner successfully added the watchdog descriptor to Confirmator on it's side. The most optimal way would be to keep the descriptor mounted on TX side until we get signalling confirmation that RX side successfully mounted the same descriptor. If transport fails (receiver crashed), TX should perform necessary cleanup. The question is how this bookkeeping and signalling should be implemented - that's what we need to think about.

For UDP, as long as UDP link is unreliable, we don't much care - UDP can loose both SHM and raw data and we don't aim to have any sort of reliability in any of these cases