Skip to content

Commit a21666b

Browse files
CCOSTANCCOSTAN
committed
Add Nebula Sync health dispatch
1 parent 4c3a8a0 commit a21666b

3 files changed

Lines changed: 177 additions & 1 deletion

File tree

‎config/packages/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ Live collection of plug-and-play Home Assistant packages. Each YAML file in this
5050
| [docker_infrastructure.yaml](docker_infrastructure.yaml) | Docker host patching telemetry, container/stack Repairs automation, 20-minute Joanna escalation for persistent container outages using stable configured monitor membership, and weekly scheduled prune actions across docker_10/14/17/69. | `sensor.docker_*_apt_status`, `binary_sensor.*_stack_status`, `sensor.docker_stacks_down_count`, `repairs.create`, `script.joanna_dispatch` |
5151
| [proxmox.yaml](proxmox.yaml) | Proxmox runtime and disk pressure monitoring with Repairs + Joanna dispatch for sustained node degradations, plus nightly Frigate reboot. | `binary_sensor.proxmox*_runtime_healthy`, `sensor.proxmox*_disk_used_percentage`, `repairs.create`, `script.joanna_dispatch`, `button.qemu_docker2_101_reboot` |
5252
| [synology_dsm.yaml](synology_dsm.yaml) | Synology DSM integration health normalization for Carlo-NAS01 and Carlo-NVR, with outage-aware Joanna-first handling for lone post-outage volume warnings and Repairs escalation for persistent or non-outage problems. | `binary_sensor.carlo_*_synology_problem`, `sensor.carlo_*_synology_problem_summary`, `binary_sensor.powerwall_grid_status`, `repairs.create`, `script.joanna_dispatch` |
53-
| [infrastructure.yaml](infrastructure.yaml) | Normalized WAN/DNS/backup/domain/cert health, Glances-backed Docker host disk pressure with Joanna-only warning cleanup and critical Repairs, and website uptime/latency SLO signals for Infrastructure dashboards, plus nightly backup verification and monthly Joanna HA log hygiene review with GitHub issue follow-up. | `sensor.docker_*_disk_used_percentage`, `automation.docker_host_disk_pressure_monitor`, `binary_sensor.infra_website_uptime_slo_breach`, `binary_sensor.infra_website_latency_degraded`, `automation.infra_backup_nightly_verification`, `script.joanna_dispatch` |
53+
| [infrastructure.yaml](infrastructure.yaml) | Normalized WAN/DNS/backup/domain/cert health, Nebula Sync primary/backup Pi-hole consistency monitoring with Joanna dispatch, Glances-backed Docker host disk pressure with Joanna-only warning cleanup and critical Repairs, and website uptime/latency SLO signals for Infrastructure dashboards, plus nightly backup verification and monthly Joanna HA log hygiene review with GitHub issue follow-up. | `sensor.infra_nebula_sync_dns_consistency`, `binary_sensor.infra_nebula_sync_degraded`, `sensor.docker_*_disk_used_percentage`, `automation.infra_nebula_sync_health_dispatch`, `automation.docker_host_disk_pressure_monitor`, `binary_sensor.infra_website_uptime_slo_breach`, `binary_sensor.infra_website_latency_degraded`, `automation.infra_backup_nightly_verification`, `script.joanna_dispatch` |
5454
| [onenote_indexer.yaml](onenote_indexer.yaml) | OneNote indexer health/status monitoring for Joanna, explicit index-health confirmation, failure-repair automation, and a daily duplicate-delete maintenance request. | `sensor.onenote_indexer_last_job_status`, `binary_sensor.onenote_indexer_last_job_successful`, `binary_sensor.onenote_indexer_index_healthy` |
5555
| [mqtt_status.yaml](mqtt_status.yaml) | Command-line MQTT broker reachability probe with Spook Repairs escalation and Joanna troubleshooting dispatch on outage. | `binary_sensor.mqtt_status_raw`, `binary_sensor.mqtt_broker_problem`, `repairs.create`, `rest_command.bearclaw_command` |
5656
| [mariadb.yaml](mariadb.yaml) | MariaDB recorder health and capacity snapshots with hourly live metrics, weekly admin/recorder polling, and stats-ready numeric sensors. | `sensor.mariadb_status`, `sensor.database_size` |

‎config/packages/infrastructure.yaml‎

Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
# Notes: Docker host root disk usage uses Glances-backed normalized sensors; raw Glances sensors are recorder/logbook-filtered.
1818
# Notes: Disk-pressure dispatch allows bounded safe cleanup of disposable caches and old generated backup artifacts, but not live data or restarts.
1919
# Notes: Warning-level Docker host disk pressure is Joanna-only; Repairs are reserved for critical pressure.
20+
# Notes: Nebula Sync DNS consistency compares primary/backup Pi-hole answers and dispatches Joanna on sustained drift or container loss.
2021
######################################################################
2122

2223
input_text:
@@ -29,6 +30,9 @@ input_text:
2930
docker_69_disk_pressure_band:
3031
name: "docker_69 disk pressure band"
3132
max: 20
33+
infra_nebula_sync_health_band:
34+
name: "Nebula Sync health band"
35+
max: 20
3236

3337
input_boolean:
3438
infra_duplicati_backup_repair_active:
@@ -65,6 +69,23 @@ command_line:
6569
command: "curl -fsS https://api.ipify.org || echo unknown"
6670
scan_interval: 900
6771

72+
- sensor:
73+
name: Infra Nebula Sync DNS Consistency
74+
unique_id: infra_nebula_sync_dns_consistency
75+
command: >-
76+
/bin/bash -c 'primary=192.168.10.10; secondary=192.168.10.14; host=GTG-PF45FK6F; fqdn=GTG-PF45FK6F.fordst.com; ip=192.168.10.117; q(){ dig +time=2 +tries=1 +short @"$1" "$2" A 2>/dev/null | tr -d "\r" | sort | tr "\n" "," | sed "s/,$//"; }; r(){ dig +time=2 +tries=1 +short @"$1" -x "$2" 2>/dev/null | tr -d "\r" | sed "s/\.$//" | sort | tr "\n" "," | sed "s/,$//"; }; p_short=$(q "$primary" "$host"); s_short=$(q "$secondary" "$host"); p_fqdn=$(q "$primary" "$fqdn"); s_fqdn=$(q "$secondary" "$fqdn"); p_rev=$(r "$primary" "$ip"); s_rev=$(r "$secondary" "$ip"); status=mismatch; if [ "$p_short" = "$ip" ] && [ "$s_short" = "$ip" ] && [ "$p_fqdn" = "$ip" ] && [ "$s_fqdn" = "$ip" ] && [ -n "$p_rev" ] && [ "$p_rev" = "$s_rev" ]; then status=ok; fi; printf "{\"status\":\"%s\",\"host\":\"%s\",\"expected_ip\":\"%s\",\"primary_short\":\"%s\",\"secondary_short\":\"%s\",\"primary_fqdn\":\"%s\",\"secondary_fqdn\":\"%s\",\"primary_reverse\":\"%s\",\"secondary_reverse\":\"%s\"}\n" "$status" "$host" "$ip" "$p_short" "$s_short" "$p_fqdn" "$s_fqdn" "$p_rev" "$s_rev"'
77+
scan_interval: 300
78+
value_template: "{{ value_json.status | default('unknown') }}"
79+
json_attributes:
80+
- host
81+
- expected_ip
82+
- primary_short
83+
- secondary_short
84+
- primary_fqdn
85+
- secondary_fqdn
86+
- primary_reverse
87+
- secondary_reverse
88+
6889
template:
6990
- sensor:
7091
- name: "Infra External IP"
@@ -210,6 +231,45 @@ template:
210231
{% set service_state = states('binary_sensor.pihole_status') %}
211232
{{ switch_state != 'on' or service_state in ['off', 'unavailable', 'unknown'] }}
212233
234+
- name: "Infra Nebula Sync Degraded"
235+
unique_id: infra_nebula_sync_degraded
236+
device_class: problem
237+
state: >-
238+
{% set dns_state = states('sensor.infra_nebula_sync_dns_consistency') | lower %}
239+
{% set portainer_known = [
240+
expand('binary_sensor.nebula_sync_status') | count > 0,
241+
expand('binary_sensor.nebula_sync_status_2') | count > 0,
242+
expand('sensor.nebula_sync_state') | count > 0,
243+
expand('sensor.nebula_sync_state_2') | count > 0,
244+
expand('switch.nebula_sync_container') | count > 0,
245+
expand('switch.nebula_sync_container_2') | count > 0
246+
] | select('equalto', true) | list | count > 0 %}
247+
{% set portainer_ok = [
248+
is_state('binary_sensor.nebula_sync_status', 'on'),
249+
is_state('binary_sensor.nebula_sync_status_2', 'on'),
250+
(states('sensor.nebula_sync_state') | lower) == 'running',
251+
(states('sensor.nebula_sync_state_2') | lower) == 'running',
252+
is_state('switch.nebula_sync_container', 'on'),
253+
is_state('switch.nebula_sync_container_2', 'on')
254+
] | select('equalto', true) | list | count > 0 %}
255+
{{ dns_state != 'ok' or (portainer_known and not portainer_ok) }}
256+
attributes:
257+
dns_consistency: "{{ states('sensor.infra_nebula_sync_dns_consistency') }}"
258+
host: "{{ state_attr('sensor.infra_nebula_sync_dns_consistency', 'host') }}"
259+
expected_ip: "{{ state_attr('sensor.infra_nebula_sync_dns_consistency', 'expected_ip') }}"
260+
primary_short: "{{ state_attr('sensor.infra_nebula_sync_dns_consistency', 'primary_short') }}"
261+
secondary_short: "{{ state_attr('sensor.infra_nebula_sync_dns_consistency', 'secondary_short') }}"
262+
primary_fqdn: "{{ state_attr('sensor.infra_nebula_sync_dns_consistency', 'primary_fqdn') }}"
263+
secondary_fqdn: "{{ state_attr('sensor.infra_nebula_sync_dns_consistency', 'secondary_fqdn') }}"
264+
primary_reverse: "{{ state_attr('sensor.infra_nebula_sync_dns_consistency', 'primary_reverse') }}"
265+
secondary_reverse: "{{ state_attr('sensor.infra_nebula_sync_dns_consistency', 'secondary_reverse') }}"
266+
nebula_status: "{{ states('binary_sensor.nebula_sync_status') }}"
267+
nebula_status_2: "{{ states('binary_sensor.nebula_sync_status_2') }}"
268+
nebula_state: "{{ states('sensor.nebula_sync_state') }}"
269+
nebula_state_2: "{{ states('sensor.nebula_sync_state_2') }}"
270+
pihole_secondary_status: "{{ states('binary_sensor.pihole_secondary_status') }}"
271+
pihole_secondary_status_2: "{{ states('binary_sensor.pihole_secondary_status_2') }}"
272+
213273
- name: "Infra UPS On Battery"
214274
unique_id: infra_ups_on_battery
215275
device_class: problem
@@ -378,6 +438,121 @@ automation:
378438
data:
379439
issue_id: infra_website_latency_degraded
380440

441+
- alias: "Infrastructure - Nebula Sync Health Dispatch"
442+
id: infra_nebula_sync_health_dispatch
443+
description: "Dispatch Joanna when Nebula Sync DNS consistency or container telemetry stays degraded."
444+
mode: queued
445+
trigger:
446+
- platform: state
447+
entity_id: binary_sensor.infra_nebula_sync_degraded
448+
to: "on"
449+
for: "00:10:00"
450+
id: degraded
451+
- platform: state
452+
entity_id: binary_sensor.infra_nebula_sync_degraded
453+
to: "off"
454+
for: "00:02:00"
455+
id: recovered
456+
- platform: homeassistant
457+
event: start
458+
id: reconcile
459+
- platform: time_pattern
460+
minutes: "/30"
461+
id: reconcile
462+
variables:
463+
issue_id: infra_nebula_sync_degraded
464+
dns_state: "{{ states('sensor.infra_nebula_sync_dns_consistency') }}"
465+
previous_band: "{{ states('input_text.infra_nebula_sync_health_band') | lower }}"
466+
degraded: "{{ is_state('binary_sensor.infra_nebula_sync_degraded', 'on') }}"
467+
nebula_status: "{{ states('binary_sensor.nebula_sync_status') }}"
468+
nebula_status_alt: "{{ states('binary_sensor.nebula_sync_status_2') }}"
469+
nebula_state: "{{ states('sensor.nebula_sync_state') }}"
470+
nebula_state_alt: "{{ states('sensor.nebula_sync_state_2') }}"
471+
pihole_secondary_status: "{{ states('binary_sensor.pihole_secondary_status') }}"
472+
pihole_secondary_status_alt: "{{ states('binary_sensor.pihole_secondary_status_2') }}"
473+
action:
474+
- choose:
475+
- conditions: "{{ degraded and previous_band != 'warning' }}"
476+
sequence:
477+
- service: repairs.remove
478+
continue_on_error: true
479+
data:
480+
issue_id: "{{ issue_id }}"
481+
- service: script.joanna_dispatch
482+
data:
483+
trigger_context: "HA automation infra_nebula_sync_health_dispatch (Infrastructure - Nebula Sync Health Dispatch)"
484+
source: "home_assistant_automation.infra_nebula_sync_health_dispatch.warning"
485+
summary: "Nebula Sync DNS consistency or container health is degraded"
486+
entity_ids:
487+
- sensor.infra_nebula_sync_dns_consistency
488+
- binary_sensor.infra_nebula_sync_degraded
489+
- binary_sensor.nebula_sync_status
490+
- binary_sensor.nebula_sync_status_2
491+
- sensor.nebula_sync_state
492+
- sensor.nebula_sync_state_2
493+
- binary_sensor.pihole_secondary_status
494+
- binary_sensor.pihole_secondary_status_2
495+
diagnostics: >-
496+
issue_id={{ issue_id }},
497+
dns_consistency={{ dns_state }},
498+
host={{ state_attr('sensor.infra_nebula_sync_dns_consistency', 'host') }},
499+
expected_ip={{ state_attr('sensor.infra_nebula_sync_dns_consistency', 'expected_ip') }},
500+
primary_short={{ state_attr('sensor.infra_nebula_sync_dns_consistency', 'primary_short') }},
501+
secondary_short={{ state_attr('sensor.infra_nebula_sync_dns_consistency', 'secondary_short') }},
502+
primary_fqdn={{ state_attr('sensor.infra_nebula_sync_dns_consistency', 'primary_fqdn') }},
503+
secondary_fqdn={{ state_attr('sensor.infra_nebula_sync_dns_consistency', 'secondary_fqdn') }},
504+
primary_reverse={{ state_attr('sensor.infra_nebula_sync_dns_consistency', 'primary_reverse') }},
505+
secondary_reverse={{ state_attr('sensor.infra_nebula_sync_dns_consistency', 'secondary_reverse') }},
506+
nebula_status={{ nebula_status }},
507+
nebula_status_2={{ nebula_status_alt }},
508+
nebula_state={{ nebula_state }},
509+
nebula_state_2={{ nebula_state_alt }},
510+
pihole_secondary_status={{ pihole_secondary_status }},
511+
pihole_secondary_status_2={{ pihole_secondary_status_alt }},
512+
primary_dns=192.168.10.10,
513+
backup_dns=192.168.10.14
514+
request: >-
515+
Investigate Nebula Sync on docker_14 and the backup Pi-hole sync path.
516+
Verify both Pi-holes answer the GTG-PF45FK6F short name, FQDN, and reverse lookup consistently.
517+
Check nebula_sync container status, Docker health, recent sync logs, and primary/replica Pi-hole API reachability.
518+
If confidence is high, perform safe remediation such as a one-time Nebula Sync run or restarting only the nebula_sync container.
519+
Do not restart Pi-hole or change DHCP/custom DNS records unless diagnostics prove data drift and the action is safe.
520+
Reply with resolved=true/false, root_cause, action_taken, verification, and next_action_required=true/false.
521+
domain_hint: ops
522+
lane_hint: joanna.ops
523+
- service: script.send_to_logbook
524+
data:
525+
topic: "DNS"
526+
message: >-
527+
Nebula Sync DNS consistency is degraded ({{ dns_state }}); Joanna investigation requested without opening a Repair.
528+
- service: input_text.set_value
529+
target:
530+
entity_id: input_text.infra_nebula_sync_health_band
531+
data:
532+
value: warning
533+
- conditions: "{{ not degraded and previous_band in ['warning', 'unavailable'] }}"
534+
sequence:
535+
- service: repairs.remove
536+
continue_on_error: true
537+
data:
538+
issue_id: "{{ issue_id }}"
539+
- service: script.send_to_logbook
540+
data:
541+
topic: "DNS"
542+
message: "Nebula Sync DNS consistency recovered; Joanna-only warning state cleared."
543+
- service: input_text.set_value
544+
target:
545+
entity_id: input_text.infra_nebula_sync_health_band
546+
data:
547+
value: normal
548+
- conditions: "{{ not degraded and previous_band not in ['normal', 'warning', 'unavailable'] }}"
549+
sequence:
550+
- service: input_text.set_value
551+
target:
552+
entity_id: input_text.infra_nebula_sync_health_band
553+
data:
554+
value: normal
555+
381556
- alias: "Docker Host Disk Pressure Monitor"
382557
id: docker_host_disk_pressure_monitor
383558
description: "Track Docker host root disk pressure from normalized Glances sensors and dispatch Joanna on band changes."

‎config/script/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@ Current automations that kick off automated resolutions (via `script.joanna_disp
5959
| `onenote_indexer_failure_open_repair` | OneNote Indexer - Open Repair On Failure | [../packages/onenote_indexer.yaml](../packages/onenote_indexer.yaml) |
6060
| `infra_backup_nightly_verification` | Infrastructure - Backup Nightly Verification | [../packages/infrastructure.yaml](../packages/infrastructure.yaml) |
6161
| `infra_monthly_log_hygiene_review` | Infrastructure - Monthly HA Log Hygiene Review | [../packages/infrastructure.yaml](../packages/infrastructure.yaml) |
62+
| `infra_nebula_sync_health_dispatch` | Infrastructure - Nebula Sync Health Dispatch | [../packages/infrastructure.yaml](../packages/infrastructure.yaml) |
6263
| `docker_state_sync_repairs_dynamic` | Docker State Sync - Repairs (Dynamic) | [../packages/docker_infrastructure.yaml](../packages/docker_infrastructure.yaml) |
6364
| `docker_group_reconcile_weekly_joanna_review` | Docker Group Reconcile - Weekly Joanna Review | [../packages/docker_infrastructure.yaml](../packages/docker_infrastructure.yaml) |
6465
| `docker_host_disk_pressure_monitor` | Docker Host Disk Pressure Monitor | [../packages/infrastructure.yaml](../packages/infrastructure.yaml) |

0 commit comments

Comments
 (0)