Skip to content

Commit 0cf0ea8

Browse files
committed
fix(chart): raise WAF floor to 5 req/s and scope to release namespace
graph-italia ingress traffic peaks at ~0.05 req/s, so the previous >1 req/s floor already silenced the alert; raise it to >5 req/s so it only fires on genuinely high sustained traffic with a 4xx storm. Also scope the query to this release's own ingresses via exported_namespace (the nginx 'namespace' label is the controller's ns, not the ingress's; the old .*graph-italia.* regex also matched the test ingresses). Now prod monitors prod, test monitors test.
1 parent f206f7a commit 0cf0ea8

1 file changed

Lines changed: 10 additions & 8 deletions

File tree

charts/graph-italia/templates/prometheusrule.yaml

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -125,25 +125,27 @@ spec:
125125
# -----------------------------------------------------------------------
126126
# Guarded by a minimum-traffic floor: at low volume a handful of scanner
127127
# 4xx (e.g. 404s) easily exceeds the 30% ratio and trips a false
128-
# positive. Requiring >1 req/s of total ingress traffic means this only
129-
# fires on a genuine high-volume 4xx storm. severity=warning so it is
130-
# not treated as a critical incident (was a noisy critical false
131-
# positive — see k8s-configuration/RIGHTSIZING.md).
128+
# positive. Requiring >5 req/s of total ingress traffic means this only
129+
# fires on a genuine high-volume 4xx storm. Scoped to this release's own
130+
# namespace via exported_namespace (the nginx `namespace` label is the
131+
# controller's ns, not the ingress's). severity=warning so it is not
132+
# treated as a critical incident (was a noisy critical false positive —
133+
# see k8s-configuration/RIGHTSIZING.md).
132134
- alert: GraphItaliaHighWAFBlocks
133135
expr: |
134136
(
135-
sum(rate(nginx_ingress_controller_requests{status=~"4..", ingress=~".*graph-italia.*"}[10m]))
137+
sum(rate(nginx_ingress_controller_requests{status=~"4..", exported_namespace="{{ .Release.Namespace }}"}[10m]))
136138
/
137-
sum(rate(nginx_ingress_controller_requests{ingress=~".*graph-italia.*"}[10m]))
139+
sum(rate(nginx_ingress_controller_requests{exported_namespace="{{ .Release.Namespace }}"}[10m]))
138140
) > 0.30
139141
and
140-
sum(rate(nginx_ingress_controller_requests{ingress=~".*graph-italia.*"}[10m])) > 1
142+
sum(rate(nginx_ingress_controller_requests{exported_namespace="{{ .Release.Namespace }}"}[10m])) > 5
141143
for: 10m
142144
labels:
143145
severity: warning
144146
service: graph-italia
145147
namespace: {{ .Release.Namespace }}
146148
annotations:
147149
summary: "High 4xx rate on Graph Italia ingress - possible attack or WAF issue"
148-
description: "{{ "{{ $value | humanizePercentage }}" }} of requests are returning 4xx (>30%) at >1 req/s for 10+ minutes"
150+
description: "{{ "{{ $value | humanizePercentage }}" }} of requests are returning 4xx (>30%) at >5 req/s for 10+ minutes"
149151
{{- end }}

0 commit comments

Comments
 (0)