Make it obvious each troubleshooting section is a recommended alert (#5046)

Reading a section on its own, nothing said it corresponded to an alert.
It opened with a bare "Metric:" line, so the threshold, for duration, and
severity existed only in the tables on the alerting page. Anyone arriving
from a firing alert's runbook_url or from search had no way to know what
condition was supposed to have triggered it.

All 20 sections now open with the full alert definition:

  **Recommended alert:** temporal_workflow_task_execution_failed with
  failure_reason=NonDeterminismError
  Fires on any occurrence, held for 1m. Routed Critical by default.
  Tune the threshold.

Two sections cover more than one alert, and say so: UNIMPLEMENTED and
INTERNAL are separate alerts on the same metric, and Workflow Task
schedule-to-start has two thresholds.

Each of the three pages also opens by saying every section on it is one of
the recommended alerts, and that if you arrived from a firing alert, the
section you landed on is its runbook.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Duncan Mackenzie <duncanma@duncanmackenzie.net>

Author: 3 people

docs/troubleshooting/execution-failures.mdx

@@ -19,7 +19,10 @@ tags:
 This guide covers failures that happen while your Workflow and Activity code is running on a Worker: replay mismatches, oversized responses, unhandled exceptions, and Local Activities that outrun the Workflow Task heartbeat timeout.
 It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service.
 
-For alert thresholds and `for` durations, see [Worker alerting](/best-practices/worker-alerting).
+Every section on this page is one of the recommended alerts in [Worker alerting](/best-practices/worker-alerting).
+Each one opens with the alert definition, explains what the condition costs you, then walks through triage.
+If you got here from a firing alert, the section you landed on is the runbook for it.
+
 For metric definitions, see the [Temporal SDK metrics reference](/references/sdk-metrics).
 
 [`temporal_workflow_task_execution_failed`](/references/sdk-metrics#workflow_task_execution_failed) carries a `failure_reason` tag along with `namespace`, `task_queue`, and `workflow_type`.
@@ -28,7 +31,8 @@ Alert on each `failure_reason` separately.
 
 ## Non-determinism error {/* #non-determinism-error */}
 
-**Metric:** `temporal_workflow_task_execution_failed` with `failure_reason=NonDeterminismError`
+**Recommended alert:** `temporal_workflow_task_execution_failed` with `failure_reason=NonDeterminismError`
+Fires on any occurrence, held for 1m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#execution-failures).
 
 Replay produced a different sequence of commands than the one recorded in Event History.
 The Worker noticed that the Workflow code it is running no longer matches what the Execution has already done.
@@ -51,7 +55,8 @@ This will not clear up on its own.
 
 ## gRPC message too large {/* #grpc-message-too-large */}
 
-**Metric:** `temporal_workflow_task_execution_failed` with `failure_reason=GrpcMessageTooLarge`
+**Recommended alert:** `temporal_workflow_task_execution_failed` with `failure_reason=GrpcMessageTooLarge`
+Fires on any occurrence, held for 1m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#execution-failures).
 
 The Workflow Task response was bigger than the gRPC message size limit.
 The Worker tried `RespondWorkflowTaskCompleted` and something rejected it: the gRPC library on the SDK side, a proxy or load balancer in the path, or the gRPC library on the Temporal Service side when it went to receive.
@@ -84,7 +89,8 @@ Check the Workflow terminate rate on your server dashboard. A spike alongside th
 
 ## Workflow Task execution failures elevated {/* #workflow-task-execution-failures-elevated */}
 
-**Metric:** `temporal_workflow_task_execution_failed` with `failure_reason=WorkflowError`
+**Recommended alert:** `temporal_workflow_task_execution_failed` with `failure_reason=WorkflowError`
+Fires when the rate goes above 10/s, held for 2m. Routed Warning by default. [Tune the threshold](/best-practices/worker-alerting#execution-failures).
 
 Workflow Tasks are failing steadily from unhandled exceptions and panics in Workflow code that the SDK catches and reports.
 `WorkflowError` is the catch-all reason. It covers thread pool exhaustion, unhandled exceptions thrown inside the Workflow function, and Data Converter errors.
@@ -108,7 +114,8 @@ Unlike `GrpcMessageTooLarge`, the Temporal Service won't terminate the Execution
 
 ## Workflow Task execution latency high {/* #workflow-task-execution-latency-high */}
 
-**Metric:** [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency), tagged `namespace`, `task_queue`, and `workflow_type`
+**Recommended alert:** [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency), tagged `namespace`, `task_queue`, and `workflow_type`
+Fires when p99 goes above 10s, held for 5m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#execution-failures).
 
 Workflow Tasks are taking too long to execute on the Worker.
 The default Workflow Task timeout is 10 seconds, so at or above that value the Temporal Service is actively timing out Workflow Tasks.
@@ -135,7 +142,8 @@ At scale this compounds: more timeouts cause more cold replays, cold replays dri
 
 ## Activity execution failures elevated {/* #activity-execution-failures-elevated */}
 
-**Metric:** [`temporal_activity_execution_failed`](/references/sdk-metrics#activity_execution_failed), tagged `activity_type`
+**Recommended alert:** [`temporal_activity_execution_failed`](/references/sdk-metrics#activity_execution_failed), tagged `activity_type`
+Fires when the rate goes above 10/s, held for 2m. Routed Warning by default. [Tune the threshold](/best-practices/worker-alerting#execution-failures).
 
 Activities are failing outright at a sustained rate: returning failures, not timing out.
 
@@ -158,7 +166,8 @@ One caveat: internal failures increment this counter no matter what category you
 
 ## Unregistered Activity invocation {/* #unregistered-activity-invocation */}
 
-**Metric:** [`temporal_unregistered_activity_invocation`](/references/sdk-metrics#unregistered_activity_invocation), tagged `activity_type`, `task_queue`, and `workflow_type`
+**Recommended alert:** [`temporal_unregistered_activity_invocation`](/references/sdk-metrics#unregistered_activity_invocation), tagged `activity_type`, `task_queue`, and `workflow_type`
+Fires on any occurrence, held for 1m. Routed Critical by default. Go SDK only. [Tune the threshold](/best-practices/worker-alerting#execution-failures).
 
 A Workflow scheduled an Activity that the Worker polling that Task Queue has no registered implementation for.
 
@@ -178,7 +187,8 @@ This is nearly always a deployment mistake rather than a runtime condition: Work
 
 ## Local Activity latency exceeds the heartbeat timeout {/* #local-activity-latency-exceeds-the-heartbeat-timeout */}
 
-**Metric:** [`temporal_local_activity_execution_latency`](/references/sdk-metrics#local_activity_execution_latency), tagged `activity_type`
+**Recommended alert:** [`temporal_local_activity_execution_latency`](/references/sdk-metrics#local_activity_execution_latency), tagged `activity_type`
+Fires when p99 goes above 30m, held for 5m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#execution-failures).
 
 A [Local Activity](/local-activity) is running past the Workflow Task heartbeat timeout, which defaults to 30 minutes.
 

docs/troubleshooting/request-failures.mdx

@@ -19,7 +19,10 @@ tags:
 This guide covers gRPC failures and slow requests between your Temporal SDK Workers or Clients and the Temporal Service.
 It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service.
 
-For alert thresholds and `for` durations, see [Worker alerting](/best-practices/worker-alerting).
+Every section on this page is one of the recommended alerts in [Worker alerting](/best-practices/worker-alerting).
+Each one opens with the alert definition, explains what the condition costs you, then walks through triage.
+If you got here from a firing alert, the section you landed on is the runbook for it.
+
 For metric definitions, see the [Temporal SDK metrics reference](/references/sdk-metrics).
 
 [`temporal_request_failure`](/references/sdk-metrics#request_failure) increments when the Temporal Service returns a non-OK gRPC status code on a standard operation.
@@ -37,7 +40,8 @@ And `status_code` values are `UPPER_SNAKE_CASE` in every SDK, matching the gRPC
 
 ## NOT_FOUND on respond operations {/* #not_found-on-respond-operations */}
 
-**Metric:** `temporal_request_failure` with `status_code=NOT_FOUND` on `RespondWorkflowTaskCompleted`, `RespondWorkflowTaskFailed`, `RespondActivityTaskCompleted`, or `RespondActivityTaskFailed`
+**Recommended alert:** `temporal_request_failure` with `status_code=NOT_FOUND` on `RespondWorkflowTaskCompleted`, `RespondWorkflowTaskFailed`, `RespondActivityTaskCompleted`, or `RespondActivityTaskFailed`
+Fires on any occurrence, held for 5m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#request-failures).
 
 A Worker finished a Workflow Task or Activity Task and reported the result, and the Temporal Service replied that the task no longer exists.
 There are three causes:
@@ -77,7 +81,8 @@ If SDK-side metrics look normal and the Execution was not terminated or timed ou
 
 ## NOT_FOUND on Activity heartbeat {/* #not_found-on-activity-heartbeat */}
 
-**Metric:** `temporal_request_failure` with `status_code=NOT_FOUND` on `RecordActivityTaskHeartbeat`
+**Recommended alert:** `temporal_request_failure` with `status_code=NOT_FOUND` on `RecordActivityTaskHeartbeat`
+Fires on any occurrence, held for 5m. Routed Warning by default, though see the caveat below. [Tune the threshold](/best-practices/worker-alerting#request-failures).
 
 A Worker heartbeated a running Activity and the Temporal Service replied that the task no longer exists.
 The Service has already cancelled the in-flight Activity Task: either the `heartbeatTimeout` fired before the next heartbeat call arrived, the `startToClose` timeout expired while the Activity was still executing, or the Workflow Execution is no longer running.
@@ -102,7 +107,8 @@ The Activity will time out on every attempt until you fix the cause, holding Tas
 
 ## RESOURCE_EXHAUSTED on user-facing operations {/* #resource_exhausted-on-user-facing-operations */}
 
-**Metric:** `temporal_request_failure` with `status_code=RESOURCE_EXHAUSTED` on `StartWorkflowExecution`, `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, `UpdateWorkflowExecution`, or `ExecuteMultiOperation`
+**Recommended alert:** `temporal_request_failure` with `status_code=RESOURCE_EXHAUSTED` on `StartWorkflowExecution`, `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, `UpdateWorkflowExecution`, or `ExecuteMultiOperation`
+Fires on any occurrence, held for 1m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#request-failures).
 
 The Temporal Service is throttling the operations your application code uses to start Workflows and deliver Signals and Updates.
 The SDK retries these automatically for up to 60 seconds.
@@ -139,7 +145,8 @@ If the cause is system overload or an open circuit breaker, the Temporal Service
 
 ## RESOURCE_EXHAUSTED on respond operations {/* #resource_exhausted-on-respond-operations */}
 
-**Metric:** `temporal_request_failure` with `status_code=RESOURCE_EXHAUSTED` on `RespondWorkflowTaskCompleted`, `RespondWorkflowTaskFailed`, `RespondActivityTaskCompleted`, or `RespondActivityTaskFailed`
+**Recommended alert:** `temporal_request_failure` with `status_code=RESOURCE_EXHAUSTED` on `RespondWorkflowTaskCompleted`, `RespondWorkflowTaskFailed`, `RespondActivityTaskCompleted`, or `RespondActivityTaskFailed`
+Fires on any occurrence, held for 5m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#request-failures).
 
 The Temporal Service is throttling Workers reporting task results.
 The SDK retries automatically, but a delayed respond call has a compounding cost: the Worker holds the Task slot until the call succeeds, and the Service-side task stays in-flight until the response lands.
@@ -165,7 +172,8 @@ Check persistence latency filtered to `UpdateWorkflowExecution`. Slow persistenc
 
 ## RESOURCE_EXHAUSTED on poll operations {/* #resource_exhausted-on-poll-operations */}
 
-**Metric:** `temporal_long_request_failure` with `status_code=RESOURCE_EXHAUSTED` on `PollWorkflowTaskQueue` or `PollActivityTaskQueue`
+**Recommended alert:** `temporal_long_request_failure` with `status_code=RESOURCE_EXHAUSTED` on `PollWorkflowTaskQueue` or `PollActivityTaskQueue`
+Fires on any occurrence, held for 5m. Routed Warning by default. [Tune the threshold](/best-practices/worker-alerting#request-failures).
 
 The Temporal Service is throttling Worker poll calls.
 Poll operations are long-poll requests, so they increment `temporal_long_request_failure` rather than `temporal_request_failure`.
@@ -193,7 +201,8 @@ Scale Worker capacity first if schedule-to-start latency is the real problem.
 
 ## UNIMPLEMENTED or INTERNAL from the Temporal Service {/* #unimplemented-or-internal-from-the-temporal-service */}
 
-**Metric:** `temporal_request_failure` with `status_code=UNIMPLEMENTED` or `status_code=INTERNAL`, on any operation
+**Recommended alert:** `temporal_request_failure` with `status_code=UNIMPLEMENTED` or `status_code=INTERNAL`, on any operation
+Two alerts, one per status code. Each fires on any occurrence, held for 2m, routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#request-failures).
 
 These two status codes point at the Temporal Service rather than at your application, and they behave differently in the SDK.
 Alert on them separately.
@@ -224,7 +233,8 @@ For UNIMPLEMENTED, check that every Frontend, History, and Matching pod is runni
 
 ## Request latency high on user-facing operations {/* #request-latency-high-on-user-facing-operations */}
 
-**Metric:** [`temporal_request_latency`](/references/sdk-metrics#request_latency) on `StartWorkflowExecution`, `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, or `ExecuteMultiOperation`
+**Recommended alert:** [`temporal_request_latency`](/references/sdk-metrics#request_latency) on `StartWorkflowExecution`, `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, or `ExecuteMultiOperation`
+Fires when p99 goes above 2s, held for 5m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#request-failures).
 
 p99 latency on the operations your application calls synchronously has risen above your threshold.
 

docs/troubleshooting/worker-capacity.mdx

@@ -21,7 +21,10 @@ tags:
 This guide covers what to do when your Workers stop keeping up with their Task Queues: slots that never free up, pollers that quit polling, Tasks sitting in the queue, and a [Sticky Execution](/sticky-execution) cache that isn't earning its keep.
 It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service.
 
-For alert thresholds and `for` durations, see [Worker alerting](/best-practices/worker-alerting).
+Every section on this page is one of the recommended alerts in [Worker alerting](/best-practices/worker-alerting).
+Each one opens with the alert definition, explains what the condition costs you, then walks through triage.
+If you got here from a firing alert, the section you landed on is the runbook for it.
+
 For Worker sizing and tuning, see [Worker performance](/develop/worker-performance) and [Worker deployment and performance](/best-practices/worker).
 
 These failure modes are chained together.
@@ -30,7 +33,8 @@ When several fire at once, don't treat them as separate problems. Work backward
 
 ## Worker Task slots exhausted {/* #worker-task-slots-exhausted */}
 
-**Metric:** [`temporal_worker_task_slots_available`](/references/sdk-metrics#worker_task_slots_available) reaching 0, tagged `worker_type`
+**Recommended alert:** [`temporal_worker_task_slots_available`](/references/sdk-metrics#worker_task_slots_available), tagged `worker_type`
+Fires when the gauge reaches 0, held for 2m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#worker-capacity).
 
 Every Task execution slot for this `worker_type` and Task Queue is occupied, so no new Tasks can be picked up.
 The SDK blocks before issuing its next poll until a slot is released.
@@ -80,7 +84,8 @@ If that goes on past the Workflow Task heartbeat timeout, 30 minutes by default,
 
 ## All pollers disconnected {/* #all-pollers-disconnected */}
 
-**Metric:** [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) reaching 0, tagged `poller_type` and `task_queue`
+**Recommended alert:** [`temporal_num_pollers`](/references/sdk-metrics#num_pollers), tagged `poller_type` and `task_queue`
+Fires when the gauge reaches 0, held for 5m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#worker-capacity).
 
 No active pollers remain for this poller type and Task Queue.
 Workers have stopped polling entirely and Tasks are accumulating with nothing to process them.
@@ -117,7 +122,8 @@ A non-zero authorization system failure count means the auth plugin itself is br
 
 ## Task completions dropped to zero {/* #task-completions-dropped-to-zero */}
 
-**Metric:** [`temporal_request`](/references/sdk-metrics#request) rate reaching 0 on `RespondWorkflowTaskCompleted` or `RespondActivityTaskCompleted`
+**Recommended alert:** [`temporal_request`](/references/sdk-metrics#request) on `RespondWorkflowTaskCompleted` or `RespondActivityTaskCompleted`
+Fires when the rate reaches 0 while the Task Queue still has demand, held for 5m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#worker-capacity).
 
 The SDK only bumps this counter after a response succeeds, so a zero rate means nothing is landing.
 
@@ -153,7 +159,8 @@ Check overall cluster health: service errors by Namespace, persistence availabil
 
 ## Workflow Task schedule-to-start latency elevated {/* #workflow-task-schedule-to-start-latency-elevated */}
 
-**Metric:** [`temporal_workflow_task_schedule_to_start_latency`](/references/sdk-metrics#workflow_task_schedule_to_start_latency), tagged `namespace` and `task_queue`
+**Recommended alert:** [`temporal_workflow_task_schedule_to_start_latency`](/references/sdk-metrics#workflow_task_schedule_to_start_latency), tagged `namespace` and `task_queue`
+Two alerts at different thresholds. One fires when p99 goes above 5s, the other above 30m. Both held for 5m and routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#worker-capacity).
 
 Workflow Tasks are sitting in the queue before a Worker picks them up.
 Alert at two levels: a lower threshold that catches meaningful delay, and a much higher one that catches a stalled Task Queue.
@@ -189,7 +196,8 @@ Then check the server-side concurrent poller count for this Task Queue to see wh
 
 ## Activity schedule-to-start latency elevated {/* #activity-schedule-to-start-latency-elevated */}
 
-**Metric:** [`temporal_activity_schedule_to_start_latency`](/references/sdk-metrics#activity_schedule_to_start_latency), tagged `namespace` and `task_queue`
+**Recommended alert:** [`temporal_activity_schedule_to_start_latency`](/references/sdk-metrics#activity_schedule_to_start_latency), tagged `namespace` and `task_queue`
+Fires when p99 goes above 30m, held for 5m. Routed Critical by default. [Tune the threshold](/best-practices/worker-alerting#worker-capacity).
 
 Activity Tasks are sitting in the queue before a Worker picks them up.
 
@@ -216,7 +224,8 @@ Check the approximate Task backlog and the server-side concurrent poller count f
 
 ## Sticky cache holding zero entries under load {/* #sticky-cache-holding-zero-entries-under-load */}
 
-**Metric:** [`temporal_sticky_cache_size`](/references/sdk-metrics#sticky_cache_size) reaching 0 while the Worker is processing Workflow Tasks
+**Recommended alert:** [`temporal_sticky_cache_size`](/references/sdk-metrics#sticky_cache_size)
+Fires when the gauge reaches 0 while the Worker is processing Workflow Tasks, held for 15m. Routed Warning by default. [Tune the threshold](/best-practices/worker-alerting#worker-capacity).
 
 This gauge tells you how many Workflow Executions the Worker is holding in its [Sticky Execution](/sticky-execution) cache right now.
 It is not the configured cache size.