You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As an Admin operating the GTFS-RT archiver,
When a redeploy or dependency change silently doubles per-fetch latency or costs,
I am alerted so I catch the regression in hours instead of days.
Acceptance Criteria
Admin sees that a Cloud Monitoring alert is configured on the GTFS-RT archiver's per-fetch latency — it fires when the median fetch time exceeds ~1.7s (≈1.5× the ~1.1s baseline), sustained ~15 minutes, and notifies some appropriate user group
Notes
Background: on Aug 5 a one-line redeploy (revert heartbeat.py #5624) rebuilt the archiver image; unpinned google-cloud-* deps drifted and per-fetch latency doubled (~1.1s → ~2.4s, ~$15 → ~$31/day, ~$480/mo). Nothing in prod watched per-fetch latency, so it lurked for days.
Add a google_monitoring_alert_policy in iac/cal-itp-data-infra/logging/us/, alongside the existing gtfs_low_write_alert; reuse the email_dds_notify channel. Watch run.googleapis.com/request_latencies p50 for service gtfs-rt-archiver.
Staging's existing SLO alert (99% under 10s) would NOT catch this — 2.4s is under 10s. The guardrail must watch the median, not the tail.
Optional second condition on run.googleapis.com/container/billable_instance_time to alert directly on the cost driver.
Cost is negligible (<$1/mo).
Complementary fixes tracked separately: pin google-cloud-* deps and reuse a module-level storage.Client().
User story / feature request
As an Admin operating the GTFS-RT archiver,
When a redeploy or dependency change silently doubles per-fetch latency or costs,
I am alerted so I catch the regression in hours instead of days.
Acceptance Criteria
Admin sees that a Cloud Monitoring alert is configured on the GTFS-RT archiver's per-fetch latency — it fires when the median fetch time exceeds ~1.7s (≈1.5× the ~1.1s baseline), sustained ~15 minutes, and notifies some appropriate user group
Notes
google-cloud-*deps drifted and per-fetch latency doubled (~1.1s → ~2.4s, ~$15 → ~$31/day, ~$480/mo). Nothing in prod watched per-fetch latency, so it lurked for days.google_monitoring_alert_policyiniac/cal-itp-data-infra/logging/us/, alongside the existinggtfs_low_write_alert; reuse theemail_dds_notifychannel. Watchrun.googleapis.com/request_latenciesp50 for servicegtfs-rt-archiver.run.googleapis.com/container/billable_instance_timeto alert directly on the cost driver.google-cloud-*deps and reuse a module-levelstorage.Client().