fix(trivy-operator): repair CVE alerts that could never fire

[ashish1099]

Fixes three defects in the trivy-operator chart's alerting, plus two documentation errors. All of them failed the same way — silently. A PromQL join that matches nothing and an alert on a metric that does not exist both look exactly like "no problems found".

Companion to #175, which records these in decisions/security-scanning.md. That PR documents them; this one fixes them.

What was broken

Defect	Effect
TrivyOperatorScannerStuck queried trivy_resource_last_scan_timestamp_seconds	The metric does not exist — trivy-operator exposes no last-scan timestamp of any kind — so the alert could never fire
ImageOutdatedAndVulnerable joined on a mismatched image key	Worked for fully-qualified refs, silently matched nothing for short-form Docker Hub refs
+ used as the join operator	Only worked because the right-hand side was filtered to 0; fragile and obscures intent
version-checker load-bearing but unmonitored	If it stopped reporting, CVE alerting went quiet with no signal
NOTES.txt advertised trivy_image_vulnerabilities_id	That metric does not exist; the real one is trivy_vulnerability_id

The join bug in detail

Trivy describes an image as three labels and the recording rule joined them into image_registry/image_repository, giving index.docker.io/library/nginx.

version-checker emits a single image label copied from the pod spec. Its urlTagSHAFromImage does pure string splitting — it strips the tag and digest and nothing else:

lastColonIndex := strings.LastIndex(image, ":")
if lastColonIndex == -1 { return image, "", "" }

So the same container is index.docker.io/library/nginx to Trivy and nginx to version-checker. The join produced no series for those images. It kept working for quay.io/...-style references, which is what made it hard to notice — the alert looked alive.

What changed

• New recording rule record::version_checker::outdated applies Docker's reference expansion rules (nginx → index.docker.io/library/nginx, bitnami/redis → index.docker.io/bitnami/redis, docker.io/… → index.docker.io/…) so both sides share a canonical form before joining. The four regexes are mutually exclusive, so chaining applies at most one transformation per series.
• Join operator is now multiplication against a count-normalised (value 1) series. The CVE count is preserved and group_left still carries latest_version into the annotation.
• TrivyOperatorScannerStuck → TrivyOperatorMetricsMissing, inferring scanner liveness from whether vulnerability metrics exist at all, since no timestamp metric is available to query.
• VersionCheckerMetricsMissing — covers the dependency that had no alerts of its own.
• TrivyVersionCheckerJoinEmpty — the interesting one. Fires when both inputs have series but their join is empty, which means the normalisation no longer matches how images are referenced in this cluster.

That last alert is the general fix for this whole class of bug: the join now monitors itself. Any future drift in image reference style surfaces as an alert instead of as silence.

Verification

• helm template renders cleanly, and the escaped Prometheus templating ({{ $labels.image }}) survives Helm's own templating
• promtool check rules — 6 rules, SUCCESS
• Grepped the repo for stale references to the removed alert and both nonexistent metrics; only the intentional explanatory notes remain
• No cluster was queried

Note for review

VersionCheckerMetricsMissing fires if trivy-operator is deployed without version-checker. That is intentional — the chart README already states version-checker is required for ImageOutdatedAndVulnerable, and #175 makes the two a standard pair. If we ever want trivy-only deployments to be silent, this needs a values toggle.

🤖 Generated with Claude Code