Skip to content

Commit 7037941

Browse files
nafsonignafsonig
committed
implemented the maching
1 parent aa46f30 commit 7037941

2 files changed

Lines changed: 435 additions & 0 deletions

File tree

‎monitoring/anomaly-detection-alert-rules.yml‎

Lines changed: 229 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,229 @@
1+
# Prometheus Alert Rules for Anomaly Detection System
2+
3+
groups:
4+
- name: anomaly_detection_alerts
5+
interval: 30s
6+
rules:
7+
# Critical Alerts
8+
- alert: HighCriticalAnomalyDetectionRate
9+
expr: |
10+
increase(anomaly_alerts_total{severity="CRITICAL"}[5m]) > 10
11+
for: 5m
12+
labels:
13+
severity: critical
14+
service: anomaly-detection
15+
annotations:
16+
summary: "High rate of critical anomalies detected"
17+
description: |
18+
{{ $value }} critical anomalies detected in the last 5 minutes.
19+
This indicates potential coordinated market manipulation.
20+
Immediate investigation and action required.
21+
22+
- alert: CriticalPatternDetected
23+
expr: |
24+
increase(anomaly_detections_total{pattern=~"SPOOFING.*|LAYERING.*"}[1m]) > 3
25+
for: 2m
26+
labels:
27+
severity: critical
28+
service: anomaly-detection
29+
annotations:
30+
summary: "Critical spoofing or layering pattern detected"
31+
description: |
32+
{{ $value }} critical spoofing/layering events detected.
33+
Pattern: {{ $labels.pattern }}
34+
Symbol: {{ $labels.symbol }}
35+
Auto-ban action should be verified.
36+
37+
- alert: AnomalyDetectionServiceDown
38+
expr: |
39+
up{job="anomaly-detection"} == 0
40+
for: 2m
41+
labels:
42+
severity: critical
43+
service: anomaly-detection
44+
annotations:
45+
summary: "Anomaly Detection Service is down"
46+
description: |
47+
The anomaly detection service has been unavailable for more than 2 minutes.
48+
Market surveillance is currently not operational.
49+
50+
# High Severity Alerts
51+
- alert: HighAnomalyDetectionRate
52+
expr: |
53+
increase(anomaly_alerts_total{severity=~"HIGH|CRITICAL"}[5m]) > 25
54+
for: 5m
55+
labels:
56+
severity: warning
57+
service: anomaly-detection
58+
annotations:
59+
summary: "Elevated rate of high/critical anomalies"
60+
description: |
61+
{{ $value }} high or critical anomalies detected in 5 minutes.
62+
Review alert dashboard and take appropriate actions.
63+
64+
- alert: HighFalsePositiveRate
65+
expr: |
66+
(increase(anomaly_alerts_resolved_as_false_positive[24h]) /
67+
increase(anomaly_alerts_total[24h])) > 0.20
68+
for: 30m
69+
labels:
70+
severity: warning
71+
service: anomaly-detection
72+
annotations:
73+
summary: "High false positive rate detected"
74+
description: |
75+
False positive rate is {{ $value | humanizePercentage }}.
76+
Review and adjust detection thresholds.
77+
Current pattern: {{ $labels.pattern }}
78+
79+
- alert: LargeNumberOfThrottledActors
80+
expr: |
81+
anomaly_throttled_traders_count > 50
82+
for: 10m
83+
labels:
84+
severity: warning
85+
service: anomaly-detection
86+
annotations:
87+
summary: "Large number of traders throttled"
88+
description: |
89+
{{ $value }} traders are currently throttled.
90+
Review alert patterns and consider threshold adjustment.
91+
92+
- alert: DetectionLatencyHigh
93+
expr: |
94+
histogram_quantile(0.95, detection_latency_ms) > 1000
95+
for: 5m
96+
labels:
97+
severity: warning
98+
service: anomaly-detection
99+
annotations:
100+
summary: "Detection latency exceeding 1 second"
101+
description: |
102+
95th percentile detection latency is {{ $value }}ms.
103+
System may be unable to react quickly to market manipulation.
104+
105+
# Medium Severity Alerts
106+
- alert: AnomalyDetectionBacklogGrowing
107+
expr: |
108+
increase(anomaly_detection_queue_depth[5m]) > 0
109+
for: 10m
110+
labels:
111+
severity: warning
112+
service: anomaly-detection
113+
annotations:
114+
summary: "Detection backlog is accumulating"
115+
description: |
116+
Detection queue depth has increased over 10 minutes.
117+
May indicate processing bottleneck or high market volatility.
118+
119+
- alert: FeatureExtractionSlowdown
120+
expr: |
121+
histogram_quantile(0.99, feature_extraction_duration_ms) > 500
122+
for: 10m
123+
labels:
124+
severity: warning
125+
service: anomaly-detection
126+
annotations:
127+
summary: "Feature extraction taking longer than expected"
128+
description: |
129+
99th percentile feature extraction time is {{ $value }}ms.
130+
Consider optimizing feature calculation or reducing patterns.
131+
132+
- alert: MLModelLatencyHigh
133+
expr: |
134+
histogram_quantile(0.95, ml_detection_latency_ms) > 2000
135+
for: 5m
136+
labels:
137+
severity: warning
138+
service: anomaly-detection
139+
annotations:
140+
summary: "ML model inference latency elevated"
141+
description: |
142+
ML model 95th percentile latency is {{ $value }}ms.
143+
May want to disable ML engine during high-load periods.
144+
145+
# Low Severity Alerts (Informational)
146+
- alert: UnusualVolumeSpikesDetected
147+
expr: |
148+
increase(anomaly_detections_total{pattern="UNUSUAL_VOLUME_SPIKE"}[1h]) > 100
149+
for: 30m
150+
labels:
151+
severity: info
152+
service: anomaly-detection
153+
annotations:
154+
summary: "Elevated number of unusual volume spike detections"
155+
description: |
156+
{{ $value }} volume spike anomalies in the last hour.
157+
This may indicate genuine market volatility or needs threshold adjustment.
158+
159+
- alert: HistoricalDataBufferFull
160+
expr: |
161+
anomaly_historical_data_size_percent > 90
162+
for: 30m
163+
labels:
164+
severity: info
165+
service: anomaly-detection
166+
annotations:
167+
summary: "Historical data buffer approaching capacity"
168+
description: |
169+
Historical data storage is at {{ $value }}% capacity.
170+
Consider archiving old data or adjusting retention period.
171+
172+
- alert: AlertResolutionBacklog
173+
expr: |
174+
anomaly_alerts_unresolved_older_than_24h > 10
175+
for: 60m
176+
labels:
177+
severity: info
178+
service: anomaly-detection
179+
annotations:
180+
summary: "Backlog of unresolved alerts"
181+
description: |
182+
{{ $value }} alerts remain unresolved for >24 hours.
183+
May need additional review resources.
184+
185+
- name: anomaly_detection_slo
186+
interval: 1m
187+
rules:
188+
# SLO: 99.9% uptime
189+
- alert: AnomalyDetectionSLOViolation
190+
expr: |
191+
(1 - (increase(up{job="anomaly-detection"}[5m]) / 5)) > 0.001
192+
for: 5m
193+
labels:
194+
severity: critical
195+
slo: "99.9%"
196+
annotations:
197+
summary: "Anomaly Detection SLO violation"
198+
description: |
199+
Service availability is below 99.9% SLO target.
200+
Current uptime: {{ (1 - $value) | humanizePercentage }}
201+
202+
# SLO: 95th percentile latency < 500ms
203+
- alert: AnomalyDetectionLatencySLOViolation
204+
expr: |
205+
histogram_quantile(0.95, detection_latency_ms) > 500
206+
for: 10m
207+
labels:
208+
severity: warning
209+
slo: "detection_latency_p95_500ms"
210+
annotations:
211+
summary: "Detection latency SLO violation"
212+
description: |
213+
95th percentile latency is {{ $value }}ms (target: <500ms).
214+
Service may need optimization or resource scaling.
215+
216+
# SLO: False positive rate < 15%
217+
- alert: AnomalyDetectionAccuracySLOViolation
218+
expr: |
219+
(increase(anomaly_alerts_resolved_as_false_positive[7d]) /
220+
increase(anomaly_alerts_total[7d])) > 0.15
221+
for: 1h
222+
labels:
223+
severity: warning
224+
slo: "fp_rate_15_percent"
225+
annotations:
226+
summary: "Anomaly detection accuracy SLO violation"
227+
description: |
228+
7-day false positive rate is {{ $value | humanizePercentage }} (target: <15%).
229+
Model tuning or threshold adjustment needed.

0 commit comments

Comments
 (0)