Reload race condition causes persistent 500 responses on `/aggregated_metrics` endpoint

[roryabraham]

Describe the bug

When fluentd receives SIGHUP (e.g. via systemctl reload fluentd.service) to reload its
configuration, worker threads restart in-place. A new worker thread hits Errno::EADDRINUSE
when trying to bind its port because the old thread has not yet fully released the socket. The
crashing thread terminates silently, leaving the fluent-plugin-prometheus HTTP server on port
24231 running but unable to collect stats from the dead thread. All subsequent requests to the
/aggregated_metrics endpoint return HTTP 500 with body
"Connection refused - connect(2) for 127.0.0.1:<port>".

To Reproduce

1. Install fluent-package with the following configuration (see Your Configuration below)
2. Start Fluentd: systemctl start fluentd
3. Send a reload signal: systemctl reload fluentd.service (sends SIGHUP)
4. Observe the warn log for EADDRINUSE on one of the worker threads
5. Request http://localhost:24231/aggregated_metrics — it returns HTTP 500 with body
   "Connection refused - connect(2) for 127.0.0.1:<port>"

Expected behavior

After a reload (SIGHUP), all worker threads should successfully rebind their ports and Fluentd should continue serving /aggregated_metrics with HTTP 200. If a thread cannot rebind, it should either retry or fall back gracefully rather than crashing and leaving the supervisor in a broken state.

Your Environment

- Fluentd version: fluentd 1.19.1 (efdc4dca81c23480c9b55e13e55de6aa925b1cf5)
- Package version: fluent-package 6.0.1
- Operating system: Ubuntu 24.04.3
- Kernel version: 6.8.0-94-generic

Your Configuration

<system>
  workers 4
</system>

<source>
  @type prometheus
  bind 0.0.0.0
  port 24231
  metrics_path /metrics
</source>

<source>
  @type prometheus_output_monitor
</source>

<source>
  @type prometheus_monitor
</source>

<source>
  @type http
  bind 127.0.0.1
  port 24224
  <parse>
    @type json
  </parse>
</source>

<match **>
  @type null
</match>

Your Error Log

2026-03-31 22:47:09 +0000 [warn]: #3  0.07s: Async::Task
      | Task may have ended with unhandled exception.
      |   Errno::EADDRINUSE: Address already in use - bind
      |   → /opt/fluent/lib/ruby/gems/3.4.0/gems/io-endpoint-0.15.2/lib/io/endpoint/wrapper.rb:152 in 'Socket#bind'
      |     /opt/fluent/lib/ruby/gems/3.4.0/gems/io-endpoint-0.15.2/lib/io/endpoint/wrapper.rb:152 in 'IO::Endpoint::Wrapper#bind'
      |     /opt/fluent/lib/ruby/gems/3.4.0/gems/io-endpoint-0.15.2/lib/io/endpoint/host_endpoint.rb:68 in 'block in IO::Endpoint::HostEndpoint#bind'
      |     /opt/fluent/lib/ruby/gems/3.4.0/gems/io-endpoint-0.15.2/lib/io/endpoint/host_endpoint.rb:67 in 'Array#each'
      |     /opt/fluent/lib/ruby/gems/3.4.0/gems/io-endpoint-0.15.2/lib/io/endpoint/host_endpoint.rb:67 in 'Enumerator#each'
      |     /opt/fluent/lib/ruby/gems/3.4.0/gems/io-endpoint-0.15.2/lib/io/endpoint/host_endpoint.rb:67 in 'Enumerable#map'
      |     /opt/fluent/lib/ruby/gems/3.4.0/gems/io-endpoint-0.15.2/lib/io/endpoint/host_endpoint.rb:67 in 'IO::Endpoint::HostEndpoint#bind'
      |     /opt/fluent/lib/ruby/gems/3.4.0/gems/async-http-0.89.0/lib/async/http/endpoint.rb:216 in 'Async::HTTP::Endpoint#bind'
      |     /opt/fluent/lib/ruby/gems/3.4.0/gems/io-endpoint-0.15.2/lib/io/endpoint/generic.rb:82 in 'IO::Endpoint::Generic#accept'
      |     /opt/fluent/lib/ruby/gems/3.4.0/gems/async-http-0.89.0/lib/async/http/server.rb:67 in 'block in Async::HTTP::Server#run'
      |     /opt/fluent/lib/ruby/gems/3.4.0/gems/async-2.24.0/lib/async/task.rb:200 in 'block in Async::Task#run'
      |     /opt/fluent/lib/ruby/gems/3.4.0/gems/async-2.24.0/lib/async/task.rb:438 in 'block in Async::Task#schedule'

Additional context

The crash leaves the fluent-plugin-prometheus /aggregated_metrics endpoint permanently
returning HTTP 500 (body: "Connection refused - connect(2) for 127.0.0.1:<port>") until
Fluentd is fully restarted. The affected thread is one of the per-worker Prometheus stats
collectors that the aggregation endpoint queries internally.

The stack trace points to io-endpoint-0.15.2 and async-http-0.89.0 — the new thread starts
its HTTP server bind before the old thread's socket is fully closed, suggesting either a missing
SO_REUSEPORT/SO_REUSEADDR option or insufficient drain time before rebinding during a
SIGHUP-triggered reload.

Our workaround is to avoid systemctl reload (SIGHUP) in favor of systemctl restart (full
stop + start), which guarantees the old process is dead and all sockets released before the new
one starts.

[Watson1978]

I'm currently trying to reproduce this issue in my local environment, but I haven't been able to trigger the race condition so far.

Could you let me know how frequently this EADDRINUSE error occurs in your Ubuntu environment? For example, does it happen 100% of the time when you run systemctl reload fluentd.service, or is it more intermittent (e.g., roughly 1 in 10 times)?

Any additional context would be very helpful for debugging!

[nathanmetcalf]

Thanks @Watson1978
We have seen it about 3 or 4 times. Explicitly, when we run systemctl reload fluentd.
It's intermittent, but hard to exactly quantify. I'll try analyize some logs to get you a better answer.

[Watson1978]

I looked into the code.
When running with multiple workers (like workers 4), the Async::HTTP::Server might need reuse_port: true and reuse_address: true options to safely bind to the same port across multiple worker processes during a graceful reload.

Here is a potential patch that might fix the issue:

diff --git a/lib/fluent/plugin_helper/http_server/server.rb b/lib/fluent/plugin_helper/http_server/server.rb
index 1fdcab39..967aec56 100644
--- a/lib/fluent/plugin_helper/http_server/server.rb
+++ b/lib/fluent/plugin_helper/http_server/server.rb
@@ -55,11 +55,11 @@ def initialize(addr:, port:, logger:, default_app: nil, tls_context: nil)
           @server_task = nil
           Console.logger = Fluent::Log::ConsoleAdapter.wrap(@logger)
 
-          opts = if tls_context
-                   { ssl_context: tls_context }
-                 else
-                   {}
-                 end
+          opts = {
+            reuse_address: true,
+            reuse_port: true
+          }
+          opts[:ssl_context] = tls_context if tls_context
           @server = Async::HTTP::Server.new(App.new(@router, @logger), Async::HTTP::Endpoint.parse(@uri, **opts))
 
           if block_given?

However, since I still cannot reproduce the EADDRINUSE error in my local environment, I am not 100% confident that this will completely solve the problem.

If possible, could you try applying this patch in your environment to see if it eliminates the race condition? Let me know the results!

[Watson1978]

To resolve this issue, we need to ensure that the http_server helper properly manages its thread lifecycle.

Currently, the missing shutdown logic allows threads to leak, which can occasionally prevent a clean transition during zero-downtime updates or reloads.

We need to add an explicit thread cleanup in the helper to release the socket and terminate the background process correctly.

diff --git a/lib/fluent/plugin_helper/http_server.rb b/lib/fluent/plugin_helper/http_server.rb
index e609e29a..cd8e8aae 100644
--- a/lib/fluent/plugin_helper/http_server.rb
+++ b/lib/fluent/plugin_helper/http_server.rb
@@ -64,7 +64,7 @@ def http_server_create_http_server(title, addr:, port:, logger:, default_app: ni
           end
 
           _block_until_http_server_start do |notify|
-            thread_create(title) do
+            @_http_server_thread = thread_create(title) do
               @_http_server.start(notify)
             end
           end
@@ -91,7 +91,7 @@ def http_server_create_https_server(title, addr:, port:, logger:, default_app: n
         end
 
         _block_until_http_server_start do |notify|
-          thread_create(title) do
+          @_http_server_thread = thread_create(title) do
             @_http_server.start(notify)
           end
         end
@@ -105,6 +105,17 @@ def stop
         super
       end
 
+      def shutdown
+        if @_http_server
+          @_http_server.stop
+        end
+        if @_http_server_thread
+          @_http_server_thread.join # or @_http_server_thread.kill
+        end
+
+        super
+      end
+
       private
 
       def _http_server_overwrite_config(config, opts)