Stream client consumers not recovering after server was down for several minutes

[bvrnAXI]

Hello,

We have a setup with a RabbitMQ server (one node) and java application (one node) that uses the stream-client, both running in docker on the same server.

After normal startup, our application produces and consumes messages to and from a rabbitmq super stream (8 partitions, two consumers, single active consumer).

When we shutdown the rabbitmq server for more than 3 minutes and then restart it (restart is always within a few seconds), the stream always become available again at the server side and the client producer seems to be ok, but the consumers do not come back up and we have to restart the java app. The logs seem to be showing reconnection attempts that fail with errors like
Error while opening connection rabbitmq-stream-consumer-54: Could not get response in 10000 ms from node glrabbitmq:5552 (client) and
reason: {request_timeout,<<"Request timeout">>}, reason: reached_max_restart_intensity, Closing connection because of timeout in state 'peer_properties_exchanged' likely due to lack of client action (server)

When testing when the server was down for about one minute, the consumers always recover and our application can start producing and consuming again without rebooting.

I can not reproduce this behaviour on my local system.

Any ideas on this behaviour would be appreciated.

Regards,
Ben

[acogoluegnes]

It is normal to have errors logged while restarting the node. The client library should keep trying until it recovers the consumers though.

We'd appreciate a standalone project with instructions to reproduce. If you can only reproduce in your environment, try to set the log level to debug for both the broker and the client (com.rabbitmq.stream logger name for the client) and share the files with us. It may be verbose though.

[bvrnAXI]

@acogoluegnes ,

I have noticed that I could get my server setup to work by adding .scheduledExecutorService(Executors.newScheduledThreadPool(16)) to the EnvironmentBuilder and vice versa I could get my local setup to fail (sometimes) by adding .scheduledExecutorService(Executors.newScheduledThreadPool(2))
It seems to have something to do with the number of partitions, threads and cpu's of the machine. (I think the default scheduledExecutorService uses as many threads as there are processors)

This lead me to the following reproducable scenario (on my local computer with 12 logical processors):
stream-test.zip

• unzip attachment
• go into folder stream-test
• docker compose up -d rabbitmq
• in another terminal run the rabbitmq-stream-perf-test tool jar with java -jar stream-perf-test-latest.jar --producers 0 --consumers 2 --stream-count 1 --super-streams --super-stream-partitions 32 --single-active-consumer --consumer-names myConsumer --uris rabbitmq-stream://localhost:5552 --consumers-by-connection 100
• in the first terminal list the streams with docker exec stream-test-rabbitmq-1 rabbitmqctl list_stream_consumer_groups and see the 32 partitions with 2 consumers each
• stop rabbit with docker compose stop rabbitmq
• wait for 5 minutes
• startup again with docker compose up -d rabbitmq
• repeat the command docker exec stream-test-rabbitmq-1 rabbitmqctl list_stream_consumer_groups several times in the next period and notice that some streams appear with 1 or 2 consumers but after a while they mostly or fully disappear

When running the same scenario with 16 partitions instead of 32, the consumers all reconnect nicely.

Regards,
Ben

[acogoluegnes]

OK, thanks for the follow-up, I'll have a look at the project.

[bvrnAXI]

@acogoluegnes ,

Did you have the chance to verify/reproduce this scenario?

Regards,
Ben

[acogoluegnes]

Not yet. I'll update this discussion when I do.