fix(agent): keep acknowledging once converged, so a lost report self-heals (v1.10.14)

The deploy report is the server's only evidence that a member node applied its
keepalived.conf, and it was sent on the write path alone. Once the rendered
config was on disk the agent took the idempotency early return every cycle and
never reported again, so a single lost report - a backend restart, a 5xx, a
network blip - left the VIP reading SYNCING with an empty "Last ack" forever
while the node was demonstrably running the right config. Nothing would ever
reconcile the two; the only escape was to change the rendered config so the
agent wrote it again, which means touching a live VIP to fix a display problem.

The agent now re-asserts its state on that path too: one request per node per
poll cycle (~2.5 min), nothing written, keepalived not reloaded.

This gap dates from the original HA/VIP work rather than this release series;
it only became visible when acknowledgements were dropped for an unrelated
reason. A test pins that both daemon copies report BEFORE the early return,
since placing it after would silently restore the old behaviour.

Verified end to end on a real HA pair: discovery, instance-based adoption of
both nodes, PENDING, Apply, agent pull, the validation gate, the hash-pinned
takeover, the acknowledgement, and retirement of the one-shot authorisation.

Author: taylanbakircioglu

README.md

@@ -2474,6 +2474,7 @@ Developed with ❤️ for the HAProxy community
 
 ## Release Notes
 
+- **v1.10.14** (2026-08-14) — **A converged node keeps acknowledging**: the deploy report is the server's only evidence that a member node applied its `keepalived.conf`, and it was sent on the write path alone. Once the rendered config was on disk the agent took the idempotency early return on every cycle and never reported again, so a **single lost report** — a backend restart, a 5xx, a network blip — left the VIP reading `SYNCING (0/n)` with an empty *Last ack* forever, while the node was demonstrably running the right config. Nothing would ever reconcile the two: the node was correct, the page was not, and the only way out was to change the rendered config so the agent wrote it again. The agent now re-asserts its state on the idempotent path too, which costs one request per node per ~2.5 minutes and touches nothing on the node — keepalived is not reloaded and the file is not rewritten. This is a long-standing gap from the original HA/VIP work, surfaced when acknowledgements were dropped for an unrelated reason in v1.10.12. Agent-script change: sync the script from Agent Management and let the agents upgrade. No schema or API change.
 - **v1.10.13** (2026-08-14) — **Agent deploy acknowledgements were silently dropped** (regression in v1.10.12, fix it before or with that release): the takeover-retirement clause added to `POST /agents/{name}/keepalived-status` in v1.10.12 reused one query placeholder for both the assignment `last_deploy_hash=$n` and the comparison inside its `CASE`. PostgreSQL deduces a type per **use**, so the same placeholder came out as `text` in one and `character varying` in the other, and asyncpg rejected the statement with `AmbiguousParameterError`. The failure was not partial: the whole UPDATE never ran, so **no member ever recorded an acknowledgement**. Every VIP sat at `SYNCING (0/n)` with an empty *Last ack*, even after the nodes had deployed the config successfully, and teardown acknowledgements were lost the same way. The hash is now bound to its own placeholder, which is only ever compared against the column and therefore unambiguous. Verified against a real PostgreSQL: both statements execute, a matching hash retires the takeover authorisation, a non-matching hash and a NULL `applied_config_hash` both leave it in place, and every case records the acknowledgement. A test now asserts every `$n` in these statements is bound exactly once and that the count matches the arguments passed. Backend only: no schema, agent or API-shape change.
 - **v1.10.12** (2026-08-14) — **A valid keepalived config is no longer rejected by its own warning**: before writing a rendered `keepalived.conf` the agent validates it with `keepalived -t` and, on failure, keeps the running config and does not restart keepalived. That fail-safe is right, but it treated **any** non-zero exit as invalid, and keepalived's config-test exit code does not separate fatal from benign. Measured on 2.2.8: a clean config exits 0, but `Truncating auth_pass to 8 characters` exits **5** and so does a missing `}` or an `Unknown keyword`. A VRRP password longer than eight characters was therefore enough to make every apply fail, including on nodes whose own running config produces the same warning and has been serving the VIP for weeks. The gate now judges the **output**: messages known to be benign are dropped and anything that remains still fails, so it fails **closed** and an unrecognised message is treated as fatal. Verified against real keepalived: a truncation warning passes while a missing brace, an unknown keyword and a `SECURITY VIOLATION` are all still refused. The agent also **reports what keepalived said** now, in the log and in the status the HA/VIP page shows; discarding it left a correct refusal with no way to act on it. Agent-script change: sync the script from Agent Management and let the agents upgrade for it to take effect. No schema or API change.
 - **v1.10.11** (2026-08-14) — **The *Adoptable* tag names the problem that actually blocks adoption**: the tag and the disabled *Adopt* button were computed separately and could disagree. A pair blocked because its peer's `keepalived.conf` could not be parsed was labelled **MASTER missing** — technically true, since the unreadable node's `state MASTER` had not been counted, but it pointed the operator at the wrong node while the real reason sat in the button's own tooltip. Both now come from one ordered decision, so the label, its colour and the tooltip always describe the condition that stops the adoption; a group held up by an unreadable or unreachable peer reads **blocked by peer**, and two MASTERs is now distinct from none. Display only: what the endpoint accepts or refuses is unchanged. On the public repo this is the first artifact carrying v1.10.4 through v1.10.10: none was released separately, because VIP adoption did not work end to end until these fixes landed.

UPGRADE_GUIDE.md

@@ -1,3 +1,26 @@
+# Upgrade Notes — v1.10.14 (a converged node keeps acknowledging)
+
+**Agent-script change, no schema change.** No `SCHEMA_VERSION` bump. After deploying, sync the
+Linux agent script from **Agent Management** and let the agents upgrade, or the fix does not
+reach the nodes.
+
+- **Symptom:** a VIP shows `SYNCING (0/n)` with an empty *Last ack* even though every member node
+  has the rendered `keepalived.conf` on disk, keepalived is running and the VIP is held.
+- **Cause:** the deploy report was sent only when the agent actually wrote the config. Once the
+  node matched, it took the idempotency early return every cycle and never reported again, so any
+  report lost in transit was never retried and the server's view stayed stale permanently.
+- **Fix:** the agent re-asserts its state on the idempotent path as well. One request per node
+  per poll cycle (~2.5 min); nothing is written and keepalived is not reloaded.
+- **Recovery is automatic.** A VIP stuck at SYNCING converges on the first poll after the agents
+  pick up the new script. No action on the nodes, no re-apply, no edit to force a rewrite.
+- **This is not new in 1.10.12.** The gap dates from the original HA/VIP work; it only became
+  visible when acknowledgements were dropped for an unrelated reason.
+
+**Rollback:** safe. Reverting restores the previous behaviour, in which a lost acknowledgement is
+never recovered.
+
+---
+
 # Upgrade Notes — v1.10.13 (deploy acknowledgements were dropped)
 
 **Backend only, no schema change.** No `SCHEMA_VERSION` bump, no agent change. If you deployed

backend/tests/test_vip_adoption_instance.py

@@ -335,6 +335,31 @@ def test_validation_failure_reports_what_keepalived_said():
     )
 
 
+def test_converged_node_keeps_acknowledging():
+    """The idempotent path must still report, or a lost ack is never recovered.
+
+    The status report is the server's ONLY evidence that a member converged, and it used to be
+    sent solely on the write path. Once the rendered config was on disk the agent took the
+    idempotency early return every cycle and never spoke again, so a single lost report - a
+    backend restart, a 5xx, a network blip - left the VIP reading SYNCING forever with an empty
+    "Last ack" while the node was demonstrably running the right config. Seen in the field after
+    acks were dropped for an unrelated reason: the node was correct, the page was not, and
+    nothing would ever reconcile them.
+    """
+    script = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text()
+    assert script.count('_kp_report "enabled" "$vip_id" "$new_hash" "already converged"') == 2, (
+        "both daemon copies must re-assert the deploy state on the idempotent path; without it "
+        "the server can never recover a lost acknowledgement"
+    )
+    # The report has to come BEFORE the early return in both copies.
+    for m in re.finditer(r'if \[\[ -n "\$cur_hash" && "\$cur_hash" == "\$would_hash" \]\]; then(.*?)fi',
+                         script, re.S):
+        body = m.group(1)
+        assert body.index("_kp_report") < body.index("return 0"), (
+            "the acknowledgement must be sent before returning, or the early return skips it"
+        )
+
+
 def test_status_ack_statements_bind_each_placeholder_once():
     """Every `$n` in the keepalived-status UPDATEs must be used exactly once, and the count must
     match the arguments passed.

backend/utils/agent_scripts/linux_install.sh

@@ -1865,6 +1865,13 @@ fetch_and_deploy_keepalived_config() {
         cur_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
         would_hash=$(printf '%s' "$new_conf" | md5sum 2>/dev/null | awk '{print $1}')
         if [[ -n "$cur_hash" && "$cur_hash" == "$would_hash" ]]; then
+            # STILL ACK. This report is the server's only evidence that the node converged, and
+            # it used to be sent on the write path alone — so a single lost ack (a backend
+            # restart, a 5xx, a network blip) left the VIP reading SYNCING forever: the node was
+            # already correct on disk, took this early return every cycle, and never spoke again.
+            # Re-asserting the state makes the loop self-healing, costs one request per ~2.5
+            # minutes, and touches nothing on the node.
+            _kp_report "enabled" "$vip_id" "$new_hash" "already converged"
             return 0
         fi
     fi
@@ -3495,6 +3502,9 @@ CONFIG_RESPONSE_EOF
             cur_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
             would_hash=$(printf '%s' "$new_conf" | md5sum 2>/dev/null | awk '{print $1}')
             if [[ -n "$cur_hash" && "$cur_hash" == "$would_hash" ]]; then
+                # See the heredoc copy: the ack must be re-asserted here or a single lost report
+                # leaves the VIP reading SYNCING forever.
+                _kp_report "enabled" "$vip_id" "$new_hash" "already converged"
                 return 0
             fi
         fi

backend/version.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.10.13",
-  "releaseName": "Agent deploy acknowledgements were silently dropped",
+  "version": "1.10.14",
+  "releaseName": "A converged node keeps acknowledging, so a lost report self-heals",
   "releaseDate": "2026-08-14"
 }

frontend/package.json

@@ -1,6 +1,6 @@
 {
   "name": "haproxy-openmanager-frontend",
-  "version": "1.10.13",
+  "version": "1.10.14",
   "description": "HAProxy Load Balancer Management UI",
   "license": "AGPL-3.0-or-later",
   "dependencies": {