fix(server): report what a timed-out stream limit change actually did

Bounding a stream that has outgrown its new limit sheds the excess before the
broker replies, so the call takes as long as the deletion. Startup gave up
first and logged that the stream was "left at its live configuration" while the
broker went on to apply the change — telling the operator the opposite of what
had happened.

Startup now waits on a handle of its own, separate from the one the health
indicator and monitor share, because the two want opposite timeouts: a probe
has to fail fast to be worth alerting on, and an admin call has to outlast the
bytes it is deleting. When it still cannot get an answer it re-reads the stream
and reports the limits actually in force rather than assuming its own update
failed.

Found by deploying #1501 to staging, where a 32.3 GB GitHub stream took longer
to shed than the client would wait.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LK2yxSEfNL5Q5xwbAqAMWr

Author: FelixTJDietrich

.changeset/webhook-limit-update-reports-the-truth.md

@@ -0,0 +1,7 @@
+---
+"hephaestus": patch
+---
+
+A webhook stream limit change that takes longer than the broker's reply now says what actually happened. Bounding a stream that has outgrown its new limit deletes the excess before the broker answers, so on a large stream the wait is proportional to the data being shed — long enough that startup gave up and reported the stream "left at its live configuration" while the broker went on to apply the change. An operator reading that was told the opposite of the truth: that nothing had happened, when tens of gigabytes had just been deleted. Startup now waits long enough for the change to land, and if it still cannot get an answer it reports the limits the stream actually has rather than assuming its own update failed.
+
+**Operators:** the new `HEPHAESTUS_WEBHOOK_STREAM_LIMIT_UPDATE_TIMEOUT` defaults to `5m` and applies only at startup. Raise it if a stream is large enough that shedding the excess takes longer than that; it is deliberately separate from the health-check timeouts, which must stay short to be worth alerting on.

docker/.env.example

@@ -128,6 +128,9 @@ HEPHAESTUS_WEBHOOK_STREAM_MAX_BYTES_GITHUB=10GB
 # deliberate start-up to apply it — for example when first bounding a stream that has already grown
 # past the new limit.
 HEPHAESTUS_WEBHOOK_STREAM_ALLOW_DESTRUCTIVE_LIMIT_UPDATES=false
+# How long one stream limit change may take. A bound that sheds messages deletes them before the
+# broker answers, so raise this on a stream too large to shed inside five minutes.
+HEPHAESTUS_WEBHOOK_STREAM_LIMIT_UPDATE_TIMEOUT=5m
 
 # -----------------------------------------------------------------------------
 # MONITORING & SYNC

docker/compose.core.yaml

@@ -42,6 +42,9 @@ services:
       # Lowering a bound on a stream that has already outgrown it deletes the excess. Set true for
       # one deliberate start-up to apply it; startup otherwise logs what it would cost and declines.
       HEPHAESTUS_WEBHOOK_STREAM_ALLOW_DESTRUCTIVE_LIMIT_UPDATES: ${HEPHAESTUS_WEBHOOK_STREAM_ALLOW_DESTRUCTIVE_LIMIT_UPDATES:-false}
+      # Applying that bound sheds the excess before the broker replies, so the wait scales with the
+      # bytes deleted. Raise it on a stream large enough that 5 minutes is not enough.
+      HEPHAESTUS_WEBHOOK_STREAM_LIMIT_UPDATE_TIMEOUT: ${HEPHAESTUS_WEBHOOK_STREAM_LIMIT_UPDATE_TIMEOUT:-5m}
       # Shared webhook secret used for incoming HMAC / token verification.
       # WEBHOOK_EXTERNAL_URL is only used by auto-registration (server role) and is intentionally
       # not set on the webhook-server pod.

docs/admin/webhook-ingestion-operations.mdx

@@ -124,6 +124,19 @@ To apply it deliberately:
 Leaving that flag set turns a one-time decision into standing permission for any future deploy to
 delete stored messages.
 
+Shedding the excess happens **before** the broker replies, so on a large stream the call takes as
+long as the deletion does. `HEPHAESTUS_WEBHOOK_STREAM_LIMIT_UPDATE_TIMEOUT` (default `5m`) is how
+long startup waits; raise it if your stream is too large to shed inside that. If it still runs out,
+startup reports the limits the stream actually has rather than assuming its own update failed —
+which matters, because a timed-out update is one the broker may well have applied:
+
+```text
+Failed to reconcile JetStream stream limits: name=github changed=[maxBytes -1 -> 10737418240]
+— live configuration is now maxAge=PT4320H maxBytes=10737418240 maxMessages=-1
+```
+
+That line says the change **did** land. Confirm with `nats stream info github` before retrying.
+
 If instead you see a shape-drift error, the stream is not what this deployment expects and no bound
 is written onto it at all:
 
@@ -213,6 +226,29 @@ retention, so a deployment offline past the stream's byte or age bound comes bac
 at messages the stream no longer holds — `webhook.stream.unacknowledged.deletions` will say so on the
 first poll.
 
+## Making a change actually reach the broker
+
+Two things in the deployment path will silently leave your change unapplied.
+
+**The webhook receiver and NATS are in the core stack, and staging CD deploys the app only.** A
+change to `hephaestus.webhook.stream.*` ships with the application server but does not reach the
+container that bootstraps the streams until core is deployed. CI warns when `docker/compose.core.yaml`
+changes; deploy it with **Deploy to Staging** and `deploy-core: true`.
+
+**Compose does not recreate NATS when only its config changes.** The broker's configuration is a
+Compose `configs:` block whose content renders to a bind-mounted file. Changing the content does not
+change the container's spec, so `docker compose up -d` reports no drift and the running broker keeps
+the config it started with — `max_payload` and `max_file` included. Force it:
+
+```bash
+docker compose -f compose.core.yaml up -d --force-recreate nats-server
+docker exec core-nats-server-1 cat /etc/nats/nats-server.conf   # confirm the value you set
+```
+
+Recreating the broker is safe for stored messages — JetStream data lives in the volume — but
+in-flight deliveries during the restart are lost, and webhooks are not redeliverable. Reconciliation
+covers `MONITORING_TIMEFRAME` days afterwards.
+
 ## Where these settings live
 
 Stream bounds are read by beans that only exist on the **webhook** runtime role. Setting them on the

scripts/check-env-roles.test.mjs

@@ -166,9 +166,12 @@ test("a compose file that yields no services is a failure, not a pass", () => {
 });
 
 test("compose defaults resolve the way an operator who sets nothing gets them", () => {
+	// biome-ignore lint/suspicious/noTemplateCurlyInString: Compose interpolation syntax under test, not a template literal
 	assert.equal(composeDefault("${HEPHAESTUS_RUNTIME_WEBHOOK_ENABLED:-false}"), "false");
+	// biome-ignore lint/suspicious/noTemplateCurlyInString: Compose interpolation syntax under test, not a template literal
 	assert.equal(composeDefault("${HEPHAESTUS_RUNTIME_WEBHOOK_ENABLED-false}"), "false");
 	assert.equal(composeDefault('"false"'), "false");
+	// biome-ignore lint/suspicious/noTemplateCurlyInString: Compose interpolation syntax under test, not a template literal
 	assert.equal(composeDefault("${WEBHOOK_SECRET}"), "");
 });
 

server/src/main/java/de/tum/cit/aet/hephaestus/core/webhook/WebhookProperties.java

@@ -162,6 +162,14 @@ public record Stream(
          * immediately, so it is a decision an operator makes rather than one a deploy makes for them.
          */
         @DefaultValue("false") boolean allowDestructiveLimitUpdates,
+        /**
+         * How long startup waits for one stream limit update. Bounding a stream that has outgrown
+         * the new limit deletes the excess before the broker answers, so the work is proportional to
+         * the bytes being shed rather than to the size of the request — tens of GB take far longer
+         * than the request timeout the health probes want. This is deliberately separate from the
+         * consumer request timeout so a slow admin call cannot slow a readiness answer.
+         */
+        @DefaultValue("5m") Duration limitUpdateTimeout,
         /** How often the stream monitor reads stream and consumer state. */
         @DefaultValue("60s") Duration monitorInterval
     ) {
@@ -216,6 +224,11 @@ public record Stream(
                 requirePositive("stream.maxBytesByStream." + e.getKey(), e.getValue());
             }
             requirePositive("stream.storageBudget", storageBudget);
+            if (limitUpdateTimeout.isZero() || limitUpdateTimeout.isNegative()) {
+                throw new IllegalArgumentException(
+                    "stream.limitUpdateTimeout must be positive, got: " + limitUpdateTimeout
+                );
+            }
             if (monitorInterval.isZero() || monitorInterval.isNegative()) {
                 throw new IllegalArgumentException("stream.monitorInterval must be positive, got: " + monitorInterval);
             }

server/src/main/java/de/tum/cit/aet/hephaestus/integration/core/webhook/WebhookJetStreamBootstrap.java

@@ -248,17 +248,39 @@ private boolean applyLimits(String name, LimitPlan plan) {
             }
             return true;
         } catch (JetStreamApiException | IOException ex) {
-            // The stream exists and publishes still land at its live limits, so this is not worth
-            // failing a deploy over — the receiver stays up and the operator gets a loud record.
+            // A timeout here says the broker did not answer in time, not that it did nothing: shedding
+            // the excess a new bound deletes happens before the reply, so the update can land while the
+            // client gives up. Report the limits the stream actually has rather than assert an outcome.
             log.error(
-                "Failed to reconcile JetStream stream limits: name={} — stream left at its live configuration",
+                "Failed to reconcile JetStream stream limits: name={} changed={} — live configuration is now {}",
                 name,
+                plan.applied,
+                describeLiveLimits(name),
                 ex
             );
             return false;
         }
     }
 
+    /**
+     * The stream's limits as the broker reports them right now, for a failure that cannot say whether
+     * its own update landed. Never throws: it runs on a path that is already handling a failure, and
+     * an unreadable broker is itself the answer.
+     */
+    private String describeLiveLimits(String name) {
+        try {
+            StreamConfiguration live = jsm.getStreamInfo(name).getConfiguration();
+            return String.format(
+                "maxAge=%s maxBytes=%d maxMessages=%d",
+                live.getMaxAge(),
+                live.getMaxBytes(),
+                live.getMaxMsgs()
+            );
+        } catch (JetStreamApiException | IOException | RuntimeException ex) {
+            return "unreadable (" + ex.getClass().getSimpleName() + ") — check the broker directly";
+        }
+    }
+
     /**
      * Fields that define what the stream <em>is</em>. Changing any of them reshapes or re-homes the
      * data, so they are reported and never written — {@code nats stream edit} with a human deciding

server/src/main/java/de/tum/cit/aet/hephaestus/integration/core/webhook/WebhookProducerBeans.java

@@ -9,9 +9,11 @@
 import io.nats.client.Connection;
 import io.nats.client.JetStream;
 import io.nats.client.JetStreamManagement;
+import io.nats.client.JetStreamOptions;
 import java.io.IOException;
 import org.springframework.beans.factory.annotation.Qualifier;
 import org.springframework.context.annotation.Bean;
+import org.springframework.context.annotation.Primary;
 
 /**
  * The NATS <em>producer</em> cluster contributed by {@link WebhookConfiguration} on the
@@ -34,11 +36,30 @@ JetStream webhookJetStream(@Qualifier("natsConnection") Connection natsConnectio
     }
 
     @Bean
+    @Primary
     JetStreamManagement webhookJetStreamManagement(@Qualifier("natsConnection") Connection natsConnection)
         throws IOException {
         return natsConnection.jetStreamManagement();
     }
 
+    /**
+     * The handle {@link WebhookJetStreamBootstrap} applies limit changes through. Separate from
+     * {@link #webhookJetStreamManagement} because the two want opposite timeouts: the health
+     * indicator and the monitor share that one and need it to fail fast, while bounding a stream
+     * that has outgrown its new limit sheds the excess before the broker answers and takes as long
+     * as the bytes require. Sharing one handle means either readiness waits on a delete or the
+     * delete reports a failure the broker is still completing.
+     */
+    @Bean
+    JetStreamManagement webhookAdminJetStreamManagement(
+        @Qualifier("natsConnection") Connection natsConnection,
+        WebhookProperties properties
+    ) throws IOException {
+        return natsConnection.jetStreamManagement(
+            JetStreamOptions.builder().requestTimeout(properties.stream().limitUpdateTimeout()).build()
+        );
+    }
+
     @Bean
     Retry webhookPublishRetry(WebhookProperties properties) {
         WebhookProperties.Publish p = properties.publish();
@@ -63,7 +84,10 @@ JetStreamPublisher jetStreamPublisher(
     }
 
     @Bean
-    WebhookJetStreamBootstrap webhookJetStreamBootstrap(JetStreamManagement jsm, WebhookProperties properties) {
+    WebhookJetStreamBootstrap webhookJetStreamBootstrap(
+        @Qualifier("webhookAdminJetStreamManagement") JetStreamManagement jsm,
+        WebhookProperties properties
+    ) {
         return new WebhookJetStreamBootstrap(jsm, properties);
     }
 

server/src/main/resources/application.yml

@@ -612,6 +612,10 @@ hephaestus:
             # messages the stream already holds is withheld and logged rather than applied. Set this
             # once, deliberately, to let that through.
             allow-destructive-limit-updates: ${HEPHAESTUS_WEBHOOK_STREAM_ALLOW_DESTRUCTIVE_LIMIT_UPDATES:false}
+            # A bound that sheds messages deletes them before the broker replies, so the wait scales
+            # with the bytes shed, not with the request. Startup-only, and deliberately not the
+            # timeout the health probes use — those have to fail fast to be worth alerting on.
+            limit-update-timeout: ${HEPHAESTUS_WEBHOOK_STREAM_LIMIT_UPDATE_TIMEOUT:5m}
             monitor-interval: 60s
         shutdown:
             # Drain budget for in-flight publishes after HTTP closes. Docker stop_grace_period

server/src/test/java/de/tum/cit/aet/hephaestus/core/webhook/WebhookPropertiesFixture.java

@@ -25,6 +25,7 @@ public static WebhookProperties.Stream stream() {
             Map.of(),
             gibibytes(12),
             false,
+            Duration.ofMinutes(5),
             Duration.ofSeconds(60)
         );
     }