feat(kubeaid-agent): alert on vulnerable images that have an upgrade available

Restores the outdated-and-vulnerable signal that was removed from the
trivy-operator chart, computed correctly this time.

The alert wants the intersection of two facts: an image carries a fixable
Critical/High finding, and a newer tag exists upstream. That overlap is the
actionable set. A CVE with no newer image is a wait, and a newer image with
no CVE is routine maintenance.

The old rule tried to join those in PromQL and could not. Trivy splits and
normalises registry and repository while version-checker copies the pod spec
verbatim, so joining means reimplementing Docker reference grammar in
chained label_replace calls. It failed silently on short-form Docker Hub
references, and the fix for that wrongly rewrote registry-local ones. The
agent already parses both sides with go-containerregistry, so the count is
computed there and exported as a metric.

One alert per cluster rather than one per image. The count is what decides
whether anyone acts, and a per-image alert across a fleet produces a list
nobody reads.

Also alerts on collection failure, using == 0 so it stays quiet on clusters
that simply do not run a scanner or Tetragon, which report -1. Collapsing
those two states would make the alert fire everywhere and get it silenced,
taking the real signal with it.

Author: ashish1099

argocd-helm-charts/kubeaid-agent/templates/prometheusrule.yaml

@@ -0,0 +1,64 @@
+{{- if and .Values.appConfig.securityPosture.enabled .Values.prometheusRule.enabled }}
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: {{ include "kubeaid-agent.fullname" . }}-security-posture
+  labels:
+    {{- include "kubeaid-agent.labels" . | nindent 4 }}
+    {{- with .Values.prometheusRule.additionalLabels }}
+    {{- toYaml . | nindent 4 }}
+    {{- end }}
+  {{- with .Values.prometheusRule.additionalAnnotations }}
+  annotations:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+spec:
+  groups:
+    - name: kubeaid-agent.security-posture
+      rules:
+        # The actionable set: an image carrying a fixable Critical/High finding
+        # that ALSO has a newer tag upstream. A CVE with no newer image is a
+        # wait, and a newer image with no CVE is routine maintenance. Only the
+        # overlap is work that can be done today and removes a known exposure.
+        #
+        # One alert per cluster, not one per image. The count is what decides
+        # whether anyone acts, and a per-image alert on a fleet produces a list
+        # nobody reads. The images themselves are in the VulnerabilityReport
+        # CRs.
+        #
+        # The correlation is computed by the agent, not by a PromQL join.
+        # Trivy splits and normalises registry and repository while
+        # version-checker copies the pod spec verbatim, so joining them in
+        # PromQL means reimplementing Docker reference grammar in
+        # label_replace, which fails silently on short-form and registry-local
+        # references.
+        - alert: ImageOutdatedAndVulnerable
+          expr: |
+            kubeaid_agent_security_posture_upgradable_vulnerable_images
+              > {{ .Values.prometheusRule.upgradableThreshold }}
+          for: {{ .Values.prometheusRule.upgradableFor }}
+          labels:
+            severity: warning
+          annotations:
+            summary: "{{ "{{ $value }}" }} images have fixable Critical/High CVEs with an upgrade available"
+            description: "These images can be moved to a newer tag today, and doing so clears a known fixable exposure. List them with: kubectl get vulnerabilityreports -A"
+
+        # Collection status, reported per component as 1 ok, 0 failed and -1
+        # not installed.
+        #
+        # The comparison is == 0 and must stay that way. Not-installed is -1
+        # precisely so this alert stays quiet on a cluster that never ran
+        # Tetragon or a scanner, while still firing when something that IS
+        # installed cannot be read -- a missing RBAC rule, an unreachable
+        # scanner. Collapsing the two states would make this fire everywhere
+        # and it would be silenced, taking the real signal with it.
+        - alert: SecurityPostureCollectionFailing
+          expr: |
+            kubeaid_agent_security_posture_collection == 0
+          for: {{ .Values.prometheusRule.collectionFailingFor }}
+          labels:
+            severity: warning
+          annotations:
+            summary: "kubeaid-agent cannot collect {{ "{{ $labels.component }}" }} security posture"
+            description: "The component is installed but unreadable, so its posture is unknown rather than clean. Check the agent's RBAC and the component's availability."
+{{- end }}

argocd-helm-charts/kubeaid-agent/values.schema.json

@@ -313,6 +313,33 @@
           "type": "string"
         }
       }
+    },
+    "prometheusRule": {
+      "description": "Alerts on the agent's security-posture metrics. Only rendered when appConfig.securityPosture.enabled is true.",
+      "type": "object",
+      "properties": {
+        "enabled": {
+          "type": "boolean"
+        },
+        "additionalLabels": {
+          "type": "object"
+        },
+        "additionalAnnotations": {
+          "type": "object"
+        },
+        "upgradableThreshold": {
+          "description": "Alert when more than this many images have both a fixable Critical/High finding and a newer tag upstream. One alert per cluster.",
+          "type": "integer"
+        },
+        "upgradableFor": {
+          "description": "How long the count must stay above the threshold before alerting.",
+          "type": "string"
+        },
+        "collectionFailingFor": {
+          "description": "How long a component must report collection failure before alerting.",
+          "type": "string"
+        }
+      }
     }
   }
 }

argocd-helm-charts/kubeaid-agent/values.yaml

@@ -162,3 +162,27 @@ serviceMonitor:
   interval: 30s
   scrapeTimeout: 10s
 
+# Alerts on the agent's own security-posture metrics. Only rendered when
+# appConfig.securityPosture.enabled is true, since without collection the
+# metrics never appear and every rule would evaluate against nothing.
+prometheusRule:
+  enabled: true
+
+  # -- Extra labels on the PrometheusRule object, for Prometheus selector
+  # matching.
+  additionalLabels: {}
+  additionalAnnotations: {}
+
+  # -- Alert when more than this many images have BOTH a fixable Critical/High
+  # finding and a newer tag available upstream. Zero means alert on any such
+  # image. One alert per cluster, not one per image.
+  upgradableThreshold: 0
+
+  # -- How long the count must stay above the threshold. Long by design: this
+  # is a backlog to work through, not an incident.
+  upgradableFor: 24h
+
+  # -- How long a component must report collection failure before alerting.
+  # Absorbs a transient API error without hiding a persistent one.
+  collectionFailingFor: 6h
+