Skip to content

Admin is alerted when the GTFS-RT archiver's per-fetch latency regresses or costs increase substantially #5655

Description

@vevetron

User story / feature request

As an Admin operating the GTFS-RT archiver,
When a redeploy or dependency change silently doubles per-fetch latency or costs,
I am alerted so I catch the regression in hours instead of days.

Acceptance Criteria

Admin sees that a Cloud Monitoring alert is configured on the GTFS-RT archiver's per-fetch latency — it fires when the median fetch time exceeds ~1.7s (≈1.5× the ~1.1s baseline), sustained ~15 minutes, and notifies some appropriate user group

Notes

  • Background: on Aug 5 a one-line redeploy (revert heartbeat.py #5624) rebuilt the archiver image; unpinned google-cloud-* deps drifted and per-fetch latency doubled (~1.1s → ~2.4s, ~$15 → ~$31/day, ~$480/mo). Nothing in prod watched per-fetch latency, so it lurked for days.
  • Add a google_monitoring_alert_policy in iac/cal-itp-data-infra/logging/us/, alongside the existing gtfs_low_write_alert; reuse the email_dds_notify channel. Watch run.googleapis.com/request_latencies p50 for service gtfs-rt-archiver.
  • Staging's existing SLO alert (99% under 10s) would NOT catch this — 2.4s is under 10s. The guardrail must watch the median, not the tail.
  • Optional second condition on run.googleapis.com/container/billable_instance_time to alert directly on the cost driver.
  • Cost is negligible (<$1/mo).
  • Complementary fixes tracked separately: pin google-cloud-* deps and reuse a module-level storage.Client().

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    reliabilityIssues related to the reliability of critical services that support DDS business processes.storyuser story

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions