Run the zone record reconcile from the playbook

Wires scripts/reconcile_zone_records.py into powerdns_setup.yaml, after the
service is up because it works through the HTTP API, and before the DNS
verification tests so those check the reconciled state. Primary only:
secondaries have api=no and receive these changes by AXFR.

templates/reconcile_desired.json.j2 renders the desired state from vars.yaml and
mirrors zone.template.j2. The two have to agree - one writes a zone at creation,
the other holds it there afterwards - so a test renders this template against
the real vars.yaml, feeds the result to the diff against a zone captured from
the production API, and requires it to propose nothing. The LUA content strings
are pinned against what production actually returns rather than regenerated from
the template, since those records are detect-only and a mismatched expectation
would fail the play on a zone that is entirely correct.

Idempotency is the property that makes this safe to run unconditionally, so it
is tested rather than assumed:

  - a clean zone produces no PATCH, no serial bump and no notify
  - applying the patch and recomputing proposes nothing, from three different
    starting states, and again on a third pass to rule out a two-cycle
    oscillation
  - a serial rewritten by SOA-EDIT-API does not read as drift, which it would
    if the reconcile compared a field PowerDNS always chooses itself
  - the rendered desired-state file is byte-stable, or the template task would
    report changed on every deploy

Two hardening changes came out of writing those. A _health string over 255 bytes
is now refused: PowerDNS would split it into chunks in the rdata, so what we
sent and what came back could never compare equal and the reconcile would write
on every run forever. And the template test mirrors Ansible's to_nice_json
exactly, sort_keys included, so it is testing the bytes Ansible will really
produce.

Reconcile failures are collected and raised after every zone is attempted, so
the report is printed before the play stops. rc 2 is LUA drift, where nothing
was written; rc 1 is an error. Neither is carried past.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: MorningLightMountain713

powerdns_setup.yaml

@@ -885,6 +885,102 @@
         hour: "0"
         minute: "0"
 
+    # Zone record reconcile
+    #
+    # The zone generation and import tasks earlier in this play are gated on the
+    # zone being ABSENT from SQLite, so everything zone.template.j2 writes -
+    # _health.* TXT, apex NS, SOA MNAME - lands once, at creation, and never
+    # again. Editing vars.yaml afterwards changes nothing while the run still
+    # reports success. That gate cannot simply be dropped, because the import is
+    # `pdnsutil load-zone`, which replaces the entire zone and would take the
+    # ~170 app A records and any in-flight _acme-challenge TXT with it.
+    #
+    # These tasks reconcile the template-owned records individually instead, on
+    # every run, and leave everything else alone. They come after the service is
+    # up because they work through the HTTP API, and run only on the primary -
+    # secondaries have api=no and receive these changes by AXFR.
+    - name: Wait for the PowerDNS API to accept connections
+      ansible.builtin.wait_for:
+        host: "{{ ansible_host }}"
+        port: 8081
+        timeout: 30
+      when: pdns_role == 'master'
+
+    - name: Install the zone record reconcile script
+      ansible.builtin.copy:
+        src: scripts/reconcile_zone_records.py
+        dest: /opt/pdns/scripts/reconcile_zone_records.py
+        owner: root
+        group: root
+        mode: "0755"
+      when: pdns_role == 'master'
+
+    - name: Render desired state for template-owned zone records
+      ansible.builtin.template:
+        src: templates/reconcile_desired.json.j2
+        dest: /run/pdns-reconcile-desired.json
+        owner: root
+        group: root
+        mode: "0600"
+      when: pdns_role == 'master'
+
+    # no_log because the API key is passed in the environment. The script itself
+    # never prints it, so the report and any error are surfaced by the two tasks
+    # below without exposing anything.
+    - name: Reconcile template-owned zone records
+      ansible.builtin.command:
+        argv: "{{ reconcile_argv + (['--dry-run'] if ansible_check_mode else []) }}"
+      vars:
+        reconcile_argv:
+          - /opt/pdns/scripts/reconcile_zone_records.py
+          - "--zone"
+          - "{{ item.domain }}"
+          - "--api-url"
+          - "http://{{ ansible_host }}:8081"
+          - "--desired"
+          - /run/pdns-reconcile-desired.json
+      environment:
+        PDNS_API_KEY: "{{ powerdns[env].api_key }}"
+      loop: "{{ powerdns[env].zone_configs }}"
+      loop_control:
+        label: "{{ item.domain }}"
+      register: reconcile_result
+      # Collected and raised below, so that every zone is attempted and the
+      # report is printed before the play stops.
+      failed_when: false
+      changed_when:
+        - reconcile_result.rc == 0
+        - (reconcile_result.stdout | from_json).changed | default(false)
+      check_mode: false
+      no_log: true
+      when: pdns_role == 'master'
+
+    - name: Report zone record reconcile
+      ansible.builtin.debug:
+        msg: "{{ item.stdout | from_json }}"
+      loop: "{{ reconcile_result.results | default([]) }}"
+      loop_control:
+        label: "{{ item.item.domain }}"
+      when:
+        - pdns_role == 'master'
+        - item.stdout | default('') != ''
+
+    # rc 2 is LUA record drift, where nothing was written and a human should
+    # look before anything is. rc 1 is an error. Neither is something to carry
+    # on past: every bug in the 2026-07-29 outage was a conditional that quietly
+    # did the wrong thing while the run reported success.
+    - name: Fail if any zone did not reconcile
+      ansible.builtin.fail:
+        msg: >-
+          Zone record reconcile failed for {{ item.item.domain }} (rc={{ item.rc }}).
+          {{ item.stdout | default('') }} {{ item.stderr | default('') }}
+      loop: "{{ reconcile_result.results | default([]) }}"
+      loop_control:
+        label: "{{ item.item.domain }}"
+      when:
+        - pdns_role == 'master'
+        - item.rc | default(0) != 0
+
     # Post-deployment verification
 
     - name: Test DNS resolution for geo-routing domain

scripts/reconcile_zone_records.py

@@ -55,6 +55,11 @@
 
 SOA_FIELDS = 7
 
+# A TXT string longer than this is split into several quoted chunks in the
+# rdata, so what we sent and what comes back would never compare equal and the
+# reconcile would report a change on every single run. Refuse instead.
+MAX_TXT_STRING = 255
+
 
 class ReconcileError(RuntimeError):
     """Anything that should fail the play rather than be worked around."""
@@ -153,10 +158,17 @@ def compute_changes(zone: str, existing: dict, desired: dict) -> list:
     # TTL is 300 in zone.template.j2 independently of the zone default, so it is
     # carried separately rather than derived from apex_ttl.
     health_ttl = int(desired.get("health_ttl", 300))
-    want_health = {
-        canonical("_health.{}.{}".format(r["region"], zone)): txt_rdata(r["content"])
-        for r in desired.get("health_records", [])
-    }
+    want_health = {}
+    for region in desired.get("health_records", []):
+        if len(region["content"]) > MAX_TXT_STRING:
+            raise ReconcileError(
+                "_health.{} content is {} bytes, over the {}-byte TXT string limit; "
+                "PowerDNS would split it into chunks and this would never converge".format(
+                    region["region"], len(region["content"]), MAX_TXT_STRING
+                )
+            )
+        name = canonical("_health.{}.{}".format(region["region"], zone))
+        want_health[name] = txt_rdata(region["content"])
 
     for name in sorted(want_health):
         rrset = existing.get((name, "TXT"))

templates/reconcile_desired.json.j2

@@ -0,0 +1,67 @@
+{#
+  Desired state for the template-owned records of every zone, consumed by
+  scripts/reconcile_zone_records.py. Keyed by canonical (trailing dot) zone name.
+
+  This mirrors zone.template.j2. The two must agree, because that template is
+  what writes a zone at creation and this is what holds it there afterwards - if
+  they disagree, a freshly created zone reports drift on its very first run.
+
+  Note health_ttl is carried separately rather than derived from default_ttl:
+  zone.template.j2 hardcodes 300 on the _health records while the apex records
+  take the zone default.
+
+  The lua_records entries are compared and never written. Their content strings
+  must match what the API returns byte for byte, including the record type
+  prefix and the inner quoting, or the reconcile fails the play on a zone that
+  is in fact correct. tests/test_reconcile_desired_template.py renders this file
+  and asserts exactly that against captured production output.
+#}
+{% set env = 'staging' if DEPLOY_ENV == 'staging' else 'production' %}
+{% set desired = {} %}
+{% for zone in powerdns[env].zone_configs %}
+{%   set tv = zone.template_vars %}
+{%   set health = [] %}
+{%   set lua = [] %}
+{%   if tv.geo_routing | default(false) %}
+{%     set _ = lua.append({
+         'name': '_config.' ~ zone.domain ~ '.',
+         'content': 'LUA "dofile(\'/opt/pdns/scripts/geo_routing.lua\')"'}) %}
+{%     set _ = lua.append({
+         'name': zone.domain ~ '.',
+         'content': 'A ";include(\'_config\'); return geoRoute()"'}) %}
+{%     for region in tv.geo_regions | default([]) %}
+{%       set _ = health.append({
+           'region': region.name,
+           'content': region.description ~ ' - ' ~ region.server ~ ' - ' ~ region.ip}) %}
+{%     endfor %}
+{%     if tv.status_endpoint | default(false) %}
+{%       set _ = lua.append({
+           'name': '_status.' ~ zone.domain ~ '.',
+           'content': 'TXT ";include(\'_config\'); return getServerStatus()"'}) %}
+{%     endif %}
+{%   endif %}
+{%   if tv.lua_routing | default(false) %}
+{%     set _ = lua.append({
+         'name': '_config.' ~ zone.domain ~ '.',
+         'content': 'LUA "dofile(\'/opt/pdns/scripts/' ~ tv.routing_script | default('app_routing.lua') ~ '\')"'}) %}
+{%     if tv.routing_function is defined %}
+{%       set _ = lua.append({
+           'name': '*.' ~ zone.domain ~ '.',
+           'content': 'CNAME ";include(\'_config\'); return ' ~ tv.routing_function ~ '(qname)"'}) %}
+{%     endif %}
+{%     if tv.debug_function is defined %}
+{%       set _ = lua.append({
+           'name': '_debug.' ~ zone.domain ~ '.',
+           'content': 'TXT ";include(\'_config\'); return ' ~ tv.debug_function ~ '(qname)"'}) %}
+{%     endif %}
+{%   endif %}
+{%   set _ = desired.update({zone.domain ~ '.': {
+       'apex_ttl': tv.default_ttl | default('3600') | int,
+       'health_ttl': 300,
+       'nameservers': powerdns[env].nameservers,
+       'soa_mname': powerdns[env].soa_nameserver,
+       'soa_rname': powerdns[env].soa_email,
+       'health_records': health,
+       'lua_records': lua}}) %}
+{% endfor %}
+{{ desired | to_nice_json }}