Drop decommissioned us-west region and verify node identity in health check

cdn-2.runonflux.io was shut down for non-payment and the hosting
provider re-leased its IP, 107.175.82.227, to another tenant. Every US
client subnet was still being handed that address:

  US-West  23.235.32.0/24   -> 107.175.82.227
  US-East  4.14.0.0/24      -> 107.175.82.227
  EU       159.195.85.0/24  -> 159.195.85.44
  Asia     180.188.197.0/24 -> 180.188.197.165

The new occupant answers 443 with a Let's Encrypt certificate for an
unrelated domain, so US visitors to images.runonflux.io and
cdn.runonflux.io got a dropped connection, and flux_configd's
fetch_params mirror failed on US nodes.

Remove the region from both the production (cdn-geo) and staging
(cdn-geodev) zone configs.

The deeper problem is that this never failed over. ifportup(443, ...)
only establishes that something accepts TCP on 443, and the new tenant
runs nginx there, so PowerDNS went on marking the node healthy
indefinitely. A server that had simply been powered off would have been
removed automatically; one replaced by a stranger's was not.

Switch to ifurlup() against a /health endpoint added to every CDN vhost
in flux-cdn-deploy, matching a distinctive token in the body. The check
now answers "is this still our node" rather than "is a socket open".

Details worth noting:

- ifurlup() takes a list of address SETS where ifportup() took a flat
  list. All candidates go in one set, so the selector still chooses
  among every healthy node.
- The probe URL is cdn.runonflux.io, not the per-zone hostname. It has
  to be a vhost every candidate serves, and cdn-geodev.runonflux.io is
  not served by cdn-1 or cdn-3 - using it would mark every node down and
  take the zone out entirely.
- timeout goes 2s -> 3s: the old check timed a TCP connect, this also
  completes a TLS handshake and an HTTP GET, and EU->Hong Kong is about
  three round trips. minimumFailures=3 still absorbs transient overruns.
- backupSelector stays pickclosest, so if every node were to fail the
  check the zone degrades to nearest-node rather than returning nothing.

Verified by rendering both environments and exercising the result under
a real Lua interpreter with the PowerDNS globals stubbed: URL, set
structure, per-function selectors, and that the shared option table is
copied rather than mutated between calls.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: MorningLightMountain713

templates/geo_routing.lua.j2

@@ -33,18 +33,54 @@ function getAllServerIPs()
 end
 
 
+-- Health check settings, shared by every routing function below.
+--
+-- ifurlup() rather than ifportup(): a port check only establishes that
+-- something is listening on 443. When cdn-2 was shut down for non-payment, its
+-- provider re-leased the IP to a tenant who also ran nginx there, so the port
+-- check passed and US traffic kept being routed to a stranger's server. Fetching
+-- a page we serve and matching a token in it asks the question that matters -
+-- is this still our node - instead of merely whether a socket is open.
+--
+-- Note the address argument differs between the two: ifportup() takes a flat
+-- list, ifurlup() takes a list of address SETS and uses the first set with an
+-- available member. All candidates go in one set here, so the selector chooses
+-- among every healthy node exactly as before.
+local HEALTH_URL = "{{ powerdns.geo_health_check.url }}"
+local HEALTH_OPTS = {
+    stringmatch = "{{ powerdns.geo_health_check.stringmatch }}",
+    -- Higher than the old port check: that timed a TCP connect, this must also
+    -- complete a TLS handshake and an HTTP GET, and EU->Hong Kong runs about
+    -- three round trips. minimumFailures below absorbs the occasional overrun.
+    timeout = 3,
+    minimumFailures = 3,         -- 3 consecutive failures before marking as down
+    interval = 15                -- Check every 15 seconds
+}
+
+-- Build the option table for a given selector without mutating the shared one.
+local function healthOpts(selector, backupSelector)
+    local opts = {}
+    for k, v in pairs(HEALTH_OPTS) do
+        opts[k] = v
+    end
+    opts.selector = selector
+    if backupSelector then
+        opts.backupSelector = backupSelector
+    end
+    return opts
+end
+
+
 -- Main geo-routing function called by PowerDNS
 function geoRoute()
     local all_ips = getAllServerIPs()
 
-    -- Health check with working parameters
-    local available_ips = ifportup(443, all_ips, {
-        timeout = 2,                 -- 2 seconds connection timeout
-        minimumFailures = 3,         -- 3 consecutive failures before marking as down
-        interval = 15,               -- Check every 15 seconds
-        selector = 'pickclosest',    -- Use geographic selection for healthy servers
-        backupSelector = 'pickclosest' -- Use geographic selection even when all appear down
-    })
+    -- pickclosest as backupSelector too, so that if every node fails the check
+    -- we still answer with the geographically nearest rather than at random.
+    -- ifurlup falls back to backupSelector instead of returning nothing, so a
+    -- broken health check degrades routing quality without taking the zone out.
+    local available_ips = ifurlup(HEALTH_URL, {all_ips},
+                                  healthOpts('pickclosest', 'pickclosest'))
 
     return available_ips
 end
@@ -54,13 +90,8 @@ end
 function geoRouteWeighted()
     local all_ips = getAllServerIPs()
 
-    -- Health check with same parameters as above
-    local available_ips = ifportup(443, all_ips, {
-        timeout = 2,
-        minimumFailures = 3,
-        interval = 15,
-        selector = 'all'       -- Return ALL healthy servers, not just one
-    })
+    -- selector 'all' returns every healthy server, not just one
+    local available_ips = ifurlup(HEALTH_URL, {all_ips}, healthOpts('all'))
 
     -- Return weighted selection based on geographic distribution
     return pickwrandom(available_ips)
@@ -79,12 +110,7 @@ end
 -- Function to get server status (for monitoring/debugging)
 function getServerStatus()
     local all_ips = getAllServerIPs()
-    local available_ips = ifportup(443, all_ips, {
-        timeout = 2,
-        minimumFailures = 3,
-        interval = 15,
-        selector = 'all'
-    })
+    local available_ips = ifurlup(HEALTH_URL, {all_ips}, healthOpts('all'))
 
     local status_parts = {}
     local up_count = 0

vars.yaml

@@ -21,6 +21,25 @@ powerdns:
     jammy: "5.0.6-1pdns.ubuntu22.04"
   cert_server_ip: "10.100.0.172"
 
+  # Health check behind the geo-routing Lua (see templates/geo_routing.lua.j2).
+  #
+  # This replaced ifportup(443, ...), which only proved that something accepted
+  # TCP on 443. When cdn-2 was shut down for non-payment its provider re-leased
+  # the IP to another tenant who also ran nginx on 443, so the check kept
+  # reporting the node healthy and US traffic was routed to a stranger's server
+  # for as long as it took someone to notice. Matching a token we serve makes the
+  # check answer "is this still our node", not merely "is a socket open".
+  geo_health_check:
+    # Probed against each candidate IP in turn, with this hostname as Host/SNI.
+    # It must be a vhost that every candidate node serves. cdn.runonflux.io (the
+    # cdn-aliases vhost) qualifies in both environments; the per-zone names do
+    # not - cdn-geodev.runonflux.io is not served by cdn-1 or cdn-3, so using it
+    # here would mark every node down and take the zone out entirely.
+    url: "https://cdn.runonflux.io/health"
+    # Token from cdn-locations.conf.j2 in flux-cdn-deploy. Deliberately
+    # distinctive so a recycled IP cannot satisfy the check by accident.
+    stringmatch: "flux-cdn-healthy"
+
   staging:
     zone_configs:
       - domain: "cdn-geodev.runonflux.io"
@@ -29,15 +48,17 @@ powerdns:
           default_ttl: "300"
           geo_routing: true
           lua_routing: false
+          # No us-west region: cdn-2.runonflux.io (107.175.82.227) was removed on
+          # 2026-07-29 after the server was shut down for non-payment and the
+          # provider re-leased its IP to another tenant. US clients now resolve
+          # to whichever of the remaining nodes pickclosest favours. Restoring a
+          # US region means adding it back here with a new address - do not
+          # reuse the old one.
           geo_regions:
             - name: "eu-central"
               description: "Germany EU"
               server: "cdn-1.runonflux.io"
               ip: "159.195.85.44"
-            - name: "us-west"
-              description: "West Coast USA"
-              server: "cdn-2.runonflux.io"
-              ip: "107.175.82.227"
             - name: "as-east"
               description: "Hong Kong Asia"
               server: "cdn-3.runonflux.io"
@@ -69,15 +90,17 @@ powerdns:
           default_ttl: "300"
           geo_routing: true
           lua_routing: false
+          # No us-west region: cdn-2.runonflux.io (107.175.82.227) was removed on
+          # 2026-07-29 after the server was shut down for non-payment and the
+          # provider re-leased its IP to another tenant. US clients now resolve
+          # to whichever of the remaining nodes pickclosest favours. Restoring a
+          # US region means adding it back here with a new address - do not
+          # reuse the old one.
           geo_regions:
             - name: "eu-central"
               description: "Germany EU"
               server: "cdn-1.runonflux.io"
               ip: "159.195.85.44"
-            - name: "us-west"
-              description: "West Coast USA"
-              server: "cdn-2.runonflux.io"
-              ip: "107.175.82.227"
             - name: "as-east"
               description: "Hong Kong Asia"
               server: "cdn-3.runonflux.io"