Publish the CDN node list as SRV records

Clients fall back to individual CDN nodes when the geo-routed name is unusable,
so that list cannot be discovered through the CDN - it would be circular. Today
it is a compile-time constant in a pinned wheel (flux-bootstrap
data_structures.py DIRECT_CDN_URLS, duplicated twice more in cdn_manager.py) and
in flux_iso_updater's backup_hosts. Retiring a node therefore needs a client
release rolled out to every Flux node, which is why cdn-2.runonflux.io - shut
down in July, its address since re-leased to another customer - is still in the
failover set of two separate consumers.

DNS is the right channel for it. It is a different failure domain from HTTP,
which is what a fallback path needs, and it is not a new dependency: the entries
in those hardcoded lists are already hostnames, so the client cannot use them
without a resolver either. Publishing the set here rather than the members is
strictly better than the status quo. It also holds up in the failure that
matters - if geo routing hands out a bad answer for cdn.runonflux.io, a plain
non-geo lookup against the same authoritative servers still resolves.

SRV rather than several A records at a well-known name, because A returns bare
addresses and connecting by IP defeats certificate verification - the very check
that would have caught cdn-2's address changing hands. An SRV target is a
hostname, so the client resolves it and verifies normally.

In the zone rather than in Cloudflare because records there are managed by
nothing, which is precisely how cdn-2's A record survived being cleaned out of
everywhere else. Here the reconcile owns them, templated from geo_regions - the
same source of truth that already drives geo routing and the _health records -
so retiring a region updates routing and the node list together and neither can
rot. Only geo zones have regions, so only they advertise a list.

One rrset per zone holding every target: a changed set is a single REPLACE, an
emptied one a single DELETE bound to that one name.

Mutation testing found a real gap while writing this. Comparing the records
within an rrset unsorted passed the whole suite, because every fixture happened
to be in sorted order already - PowerDNS promises no ordering, so that would have
rewritten the rrset on every run. Both SRV and the apex NS set now have a test
with deliberately unsorted input, and each fails if its sort is removed. Note
"pdns-b." sorts before "pdns." there; listing them the other way round proves
nothing.

The first production run will create the SRV rrset and change nothing else,
which is asserted directly rather than left to be discovered on deploy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: MorningLightMountain713

scripts/reconcile_zone_records.py

@@ -20,6 +20,7 @@
   written    apex NS
              apex SOA MNAME and RNAME  (never the serial)
              _health.<region> TXT
+             _cdn._tcp SRV  (the directly-reachable node list)
 
   detected   LUA records - compared and reported, never written
 
@@ -190,6 +191,45 @@ def compute_changes(zone: str, existing: dict, desired: dict) -> list:
             continue
         patch.append({"name": name, "type": "TXT", "changetype": "DELETE"})
 
+    # --- _cdn._tcp SRV: the directly-reachable node list ----------------------
+    # Clients fall back to individual nodes when the geo-routed name is unusable,
+    # so that list cannot be discovered through the CDN itself - it would be
+    # circular. It lives in DNS instead, which is a different failure domain from
+    # HTTP and one the client already depends on: the per-node names it uses today
+    # are themselves hostnames, so publishing the *set* here adds no new
+    # dependency. If geo routing hands out a bad answer, this plain non-geo lookup
+    # against the same authoritative servers still resolves.
+    #
+    # SRV rather than several A records at a well-known name, because A returns
+    # bare addresses and connecting by IP defeats TLS verification - which is the
+    # thing that would have caught cdn-2's address being re-leased. An SRV target
+    # is a hostname, so the client resolves it and verifies the certificate
+    # normally.
+    #
+    # One rrset per zone holding every target, so a changed set is a single
+    # REPLACE and an emptied one is a single DELETE.
+    srv_name = canonical("_cdn._tcp.{}".format(zone))
+    srv_ttl = int(desired.get("srv_ttl", apex_ttl))
+    want_srv = sorted(
+        "{} {} {} {}".format(
+            target.get("priority", 0),
+            target.get("weight", 5),
+            target.get("port", 443),
+            canonical(target["server"]),
+        )
+        for target in desired.get("srv_targets", [])
+    )
+    srv_rrset = existing.get((srv_name, "SRV"))
+    have_srv = sorted(r["content"] for r in srv_rrset["records"]) if srv_rrset else []
+
+    if want_srv:
+        if have_srv != want_srv or (srv_rrset is not None and srv_rrset["ttl"] != srv_ttl):
+            patch.append(_replace(srv_name, "SRV", srv_ttl, want_srv))
+    elif srv_rrset is not None:
+        # No targets configured - the zone should not advertise a node list at
+        # all. Scoped to this one name, so nothing else can be caught by it.
+        patch.append({"name": srv_name, "type": "SRV", "changetype": "DELETE"})
+
     return patch
 
 

templates/reconcile_desired.json.j2

@@ -22,6 +22,7 @@
 {%   set tv = zone.template_vars %}
 {%   set health = [] %}
 {%   set lua = [] %}
+{%   set srv = [] %}
 {%   if tv.geo_routing | default(false) %}
 {%     set _ = lua.append({
          'name': '_config.' ~ zone.domain ~ '.',
@@ -33,6 +34,16 @@
 {%       set _ = health.append({
            'region': region.name,
            'content': region.description ~ ' - ' ~ region.server ~ ' - ' ~ region.ip}) %}
+{#       The same regions, published as the directly-reachable node list. All
+         equal priority and weight: this is a fallback set for when the
+         geo-routed name is unusable, so there is no ordering to express - the
+         client picks. region.server already holds the per-node hostname, which
+         is what an SRV target must be. #}
+{%       set _ = srv.append({
+           'server': region.server,
+           'priority': 0,
+           'weight': 5,
+           'port': 443}) %}
 {%     endfor %}
 {%     if tv.status_endpoint | default(false) %}
 {%       set _ = lua.append({
@@ -58,10 +69,12 @@
 {%   set _ = desired.update({zone.domain ~ '.': {
        'apex_ttl': tv.default_ttl | default('3600') | int,
        'health_ttl': 300,
+       'srv_ttl': tv.default_ttl | default('3600') | int,
        'nameservers': powerdns[env].nameservers,
        'soa_mname': powerdns[env].soa_nameserver,
        'soa_rname': powerdns[env].soa_email,
        'health_records': health,
+       'srv_targets': srv,
        'lua_records': lua}}) %}
 {% endfor %}
 {{ desired | to_nice_json }}

tests/test_reconcile_desired_template.py

@@ -147,13 +147,44 @@ def test_ttls_follow_the_zone_template():
     assert production["app.runonflux.io."]["apex_ttl"] == 3600
 
 
+def test_first_deploy_creates_only_the_srv_rrset():
+    """cdn-geo as captured on 2026-07-30, before SRV records existed.
+
+    Publishing them is the one change the next production run should make. If
+    this ever grows a second entry, something else has drifted and wants looking
+    at before deploying.
+    """
+    zone_name = "cdn-geo.runonflux.io."
+    patch = compute_changes(
+        zone_name,
+        index_rrsets(_captured_production_zone(with_srv=False)),
+        render("production")[zone_name],
+    )
+    assert len(patch) == 1
+    assert patch[0]["name"] == "_cdn._tcp." + zone_name
+    assert patch[0]["type"] == "SRV"
+    assert sorted(r["content"] for r in patch[0]["records"]) == [
+        "0 5 443 cdn-1.runonflux.io.",
+        "0 5 443 cdn-3.runonflux.io.",
+    ]
+
+
 def test_round_trip_against_production_zone_is_a_no_op():
     """The end-to-end property: template and script agree on a live zone.
 
-    This zone was captured from the production API. Reconciling it against the
-    rendered desired state must propose nothing.
+    The steady state once SRV is published - reconciling must propose nothing.
     """
     zone_name = "cdn-geo.runonflux.io."
+    desired = render("production")[zone_name]
+    assert compute_changes(
+        zone_name, index_rrsets(_captured_production_zone(with_srv=True)), desired
+    ) == []
+
+
+def _captured_production_zone(with_srv):
+    """cdn-geo as the production API returns it, optionally including the SRV
+    rrset this change introduces."""
+    zone_name = "cdn-geo.runonflux.io."
     captured = {
         "rrsets": [
             {
@@ -200,5 +231,16 @@ def test_round_trip_against_production_zone_is_a_no_op():
             },
         ]
     }
-    desired = render("production")[zone_name]
-    assert compute_changes(zone_name, index_rrsets(captured), desired) == []
+    if with_srv:
+        captured["rrsets"].append(
+            {
+                "name": "_cdn._tcp." + zone_name,
+                "type": "SRV",
+                "ttl": 300,
+                "records": [
+                    {"content": "0 5 443 cdn-1.runonflux.io.", "disabled": False},
+                    {"content": "0 5 443 cdn-3.runonflux.io.", "disabled": False},
+                ],
+            }
+        )
+    return captured

tests/test_reconcile_zone_records.py

@@ -72,6 +72,15 @@ def geo_zone_data():
                     }
                 ],
             },
+            {
+                "name": "_cdn._tcp." + GEO_ZONE,
+                "type": "SRV",
+                "ttl": 300,
+                "records": [
+                    {"content": "0 5 443 cdn-1.runonflux.io.", "disabled": False},
+                    {"content": "0 5 443 cdn-3.runonflux.io.", "disabled": False},
+                ],
+            },
             {
                 "name": "_config." + GEO_ZONE,
                 "type": "LUA",
@@ -102,6 +111,11 @@ def geo_desired():
     return {
         "apex_ttl": 300,
         "health_ttl": 300,
+        "srv_ttl": 300,
+        "srv_targets": [
+            {"server": "cdn-1.runonflux.io", "priority": 0, "weight": 5, "port": 443},
+            {"server": "cdn-3.runonflux.io", "priority": 0, "weight": 5, "port": 443},
+        ],
         "nameservers": ["pdns.runonflux.io."],
         "soa_mname": "pdns.runonflux.io.",
         "soa_rname": "hostmaster.runonflux.io.",
@@ -494,6 +508,157 @@ def test_undotted_vars_still_compare_equal():
     assert compute_changes(GEO_ZONE, index_rrsets(geo_zone_data()), desired) == []
 
 
+# --------------------------------------------------------------------------
+# _cdn._tcp SRV - the directly-reachable node list
+# --------------------------------------------------------------------------
+
+
+def test_srv_matching_vars_produces_no_change():
+    assert compute_changes(GEO_ZONE, index_rrsets(geo_zone_data()), geo_desired()) == []
+
+
+def test_retired_node_is_dropped_from_the_srv_set():
+    """The property that makes a client release unnecessary to retire a node."""
+    zone = geo_zone_data()
+    for rrset in zone["rrsets"]:
+        if rrset["type"] == "SRV":
+            rrset["records"].append(
+                {"content": "0 5 443 cdn-2.runonflux.io.", "disabled": False}
+            )
+
+    patch = compute_changes(GEO_ZONE, index_rrsets(zone), geo_desired())
+
+    assert len(patch) == 1
+    assert patch[0]["type"] == "SRV"
+    assert patch[0]["changetype"] == "REPLACE"
+    contents = [r["content"] for r in patch[0]["records"]]
+    assert contents == ["0 5 443 cdn-1.runonflux.io.", "0 5 443 cdn-3.runonflux.io."]
+    assert not any("cdn-2" in c for c in contents)
+
+
+def test_missing_srv_rrset_is_created():
+    zone = geo_zone_data()
+    zone["rrsets"] = [r for r in zone["rrsets"] if r["type"] != "SRV"]
+    patch = compute_changes(GEO_ZONE, index_rrsets(zone), geo_desired())
+    assert len(patch) == 1
+    assert patch[0]["name"] == "_cdn._tcp." + GEO_ZONE
+    assert patch[0]["ttl"] == 300
+
+
+def test_srv_removed_entirely_when_no_targets():
+    desired = geo_desired()
+    desired["srv_targets"] = []
+    patch = compute_changes(GEO_ZONE, index_rrsets(geo_zone_data()), desired)
+    assert patch == [
+        {"name": "_cdn._tcp." + GEO_ZONE, "type": "SRV", "changetype": "DELETE"}
+    ]
+
+
+def test_srv_target_without_trailing_dot_still_compares_equal():
+    """vars.yaml holds bare hostnames; the API canonicalises. A mismatch here
+    would rewrite the rrset on every single run."""
+    desired = geo_desired()
+    for target in desired["srv_targets"]:
+        assert not target["server"].endswith(".")
+    assert compute_changes(GEO_ZONE, index_rrsets(geo_zone_data()), desired) == []
+
+
+def test_srv_target_order_in_vars_does_not_matter():
+    desired = geo_desired()
+    desired["srv_targets"].reverse()
+    assert compute_changes(GEO_ZONE, index_rrsets(geo_zone_data()), desired) == []
+
+
+def test_srv_record_order_from_the_api_does_not_matter():
+    """PowerDNS does not promise an order for the records within an rrset.
+
+    Comparing them unsorted would rewrite the rrset on every run - and every
+    fixture here happens to be pre-sorted, so nothing else would notice.
+    """
+    zone = geo_zone_data()
+    for rrset in zone["rrsets"]:
+        if rrset["type"] == "SRV":
+            rrset["records"].reverse()
+    assert compute_changes(GEO_ZONE, index_rrsets(zone), geo_desired()) == []
+
+
+def test_ns_record_order_from_the_api_does_not_matter():
+    """Same property for the apex NS set, which is compared the same way.
+
+    Note "pdns-b." sorts before "pdns." ('-' is 0x2D, '.' is 0x2E), so the
+    records below are deliberately in unsorted order - listing them the other way
+    round would be pre-sorted and would prove nothing.
+    """
+    zone = geo_zone_data()
+    desired = geo_desired()
+    desired["nameservers"] = ["pdns.runonflux.io.", "pdns-b.runonflux.io."]
+    for rrset in zone["rrsets"]:
+        if rrset["type"] == "NS":
+            rrset["records"] = [
+                {"content": "pdns.runonflux.io.", "disabled": False},
+                {"content": "pdns-b.runonflux.io.", "disabled": False},
+            ]
+    assert compute_changes(GEO_ZONE, index_rrsets(zone), desired) == []
+
+
+def test_srv_deletion_cannot_reach_another_name():
+    """The DELETE branch is bound to the single _cdn._tcp name for this zone."""
+    zone = geo_zone_data()
+    zone["rrsets"].append(
+        {
+            "name": "_sip._tcp." + GEO_ZONE,
+            "type": "SRV",
+            "ttl": 300,
+            "records": [{"content": "0 5 5060 pbx.example.com.", "disabled": False}],
+        }
+    )
+    desired = geo_desired()
+    desired["srv_targets"] = []
+    patch = compute_changes(GEO_ZONE, index_rrsets(zone), desired)
+    assert [(r["name"], r["type"]) for r in patch] == [
+        ("_cdn._tcp." + GEO_ZONE, "SRV")
+    ]
+
+
+def test_app_zone_gets_no_srv():
+    """Only geo zones have regions, so only they advertise a node list."""
+    zone = {
+        "name": APP_ZONE,
+        "rrsets": [
+            {
+                "name": APP_ZONE,
+                "type": "SOA",
+                "ttl": 3600,
+                "records": [
+                    {
+                        "content": (
+                            "pdns.runonflux.io. hostmaster.runonflux.io. "
+                            "2026073722 3600 600 86400 3600"
+                        ),
+                        "disabled": False,
+                    }
+                ],
+            },
+            {
+                "name": APP_ZONE,
+                "type": "NS",
+                "ttl": 3600,
+                "records": [{"content": "pdns.runonflux.io.", "disabled": False}],
+            },
+        ],
+    }
+    desired = {
+        "apex_ttl": 3600,
+        "nameservers": ["pdns.runonflux.io."],
+        "soa_mname": "pdns.runonflux.io.",
+        "soa_rname": "hostmaster.runonflux.io.",
+        "health_records": [],
+        "srv_targets": [],
+        "lua_records": [],
+    }
+    assert compute_changes(APP_ZONE, index_rrsets(zone), desired) == []
+
+
 # --------------------------------------------------------------------------
 # Convergence. A reconcile that runs on every deploy must settle after one
 # pass - otherwise it bumps the serial and notifies the secondaries forever.