Fix status_code casing, document the resource-exhausted metrics, restore alert 27

Audit pass. Verified every metric name, tag, tag value, SDK option, and
described behavior against SDK and server source rather than against the
source material, which was wrong on several of them.

Blocking fixes:

- status_code values are UPPER_SNAKE_CASE in every SDK. The claim that Go
  reports PascalCase (NotFound) was wrong and had reached the metrics
  reference: Go's canonicalString returns "NOT_FOUND"
  (sdk-go internal/common/metrics/tags.go) and Java uses code.name()
  (sdk-java GrpcMetricsInterceptor). A Go reader filtering
  status_code="NotFound" would have matched nothing.
- yarn check:metrics failed on the previous commit. status_code was added
  to two metrics without being added to the checker's tag allowlist or to
  the shared tag list on the page. Both now updated, along with cause.

Reference additions and corrections:

- Documented request_resource_exhausted and long_request_resource_exhausted
  (Go), which carry a cause tag naming the reason the Temporal Service
  rejected the request. Both were in bin/metrics-baseline.json with an
  empty note, which that file defines as "not yet reviewed"; removed the
  entries per its instructions. The three throttling sections now point at
  this metric instead of asking the reader to identify the cause with no
  means to do so.
- poller_type value is workflow_sticky_task, not sticky_workflow_task
  (sdk-go constants.go, sdk-java PollerTypeMetricsTag).

Scope, checked against the upstream alert set:

- Restored alert 27, Unregistered Activity invocation, which had been
  dropped entirely rather than deferred as the previous PR description
  claimed. Go SDK only, and labelled as such.
- Restored Critical severity on Workflow Task schedule-to-start and a 1m
  for duration on non-determinism error, both of which had been changed
  without a factual basis.
- Sticky cache keeps its 15m for duration, which is a deliberate deviation:
  the gauge is occupancy, so a shorter window fires on idle Workers.

Cross-page consistency:

- Both schedule-to-start sections now explain why performance-bottlenecks
  gives a lower threshold for the same metric (investigation, not paging)
  and link the ready-made queries in cloud/worker-health.
- Softened the claim that every SDK supports the BENIGN failure category,
  which was verified in Go only, and noted that internal failures
  increment activity_execution_failed regardless of category.

Verified: yarn build exits 0 with onBrokenLinks and onBrokenAnchors both
set to throw; check:metrics, check:metrics:sdks, and check:orphans pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: austonli

bin/check-metrics-reference.js

@@ -27,12 +27,14 @@ const SDKS = ['Core', 'Go', 'Java'];
 // against this, so a hyphen/underscore slip in either place is a failure.
 const TAGS = [
   'activity_type',
+  'cause',
   'failure_reason',
   'namespace',
   'nexus_operation',
   'nexus_service',
   'operation',
   'poller_type',
+  'status_code',
   'task_queue',
   'worker_type',
   'workflow_type',

bin/metrics-baseline.json

@@ -26,16 +26,6 @@
       "sdks": "Go, Java",
       "note": "Deprecated in both SDKs. Superseded by local_activity_execution_failed."
     },
-    {
-      "name": "long_request_resource_exhausted",
-      "sdks": "Go",
-      "note": ""
-    },
-    {
-      "name": "request_resource_exhausted",
-      "sdks": "Go",
-      "note": ""
-    },
     {
       "name": "resource_slots_cpu_pid_output",
       "sdks": "Core, Java",

docs/best-practices/sdk-worker-alerting.mdx

@@ -94,7 +94,7 @@ Triage guidance: [SDK Worker capacity](/troubleshooting/sdk-worker-capacity).
 | [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) | `worker_task_slots_available` | `worker_type` in `WorkflowWorker`, `ActivityWorker`, `LocalActivityWorker` | Reaches 0 | 2m | Critical |
 | [All pollers disconnected](/troubleshooting/sdk-worker-capacity#all-pollers-disconnected) | `num_pollers` | `poller_type` in `workflow_task`, `workflow_sticky_task`, `activity_task` | Reaches 0 | 5m | Critical |
 | [Task completions dropped to zero](/troubleshooting/sdk-worker-capacity#task-completions-dropped-to-zero) | `request` | `operation` in `RespondWorkflowTaskCompleted`, `RespondActivityTaskCompleted` | Rate reaches 0 while the Task Queue has demand | 5m | Critical |
-| [Workflow Task schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#workflow-task-schedule-to-start-latency-elevated) | `workflow_task_schedule_to_start_latency` | `task_queue` | p99 above 5s | 5m | Warning |
+| [Workflow Task schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#workflow-task-schedule-to-start-latency-elevated) | `workflow_task_schedule_to_start_latency` | `task_queue` | p99 above 5s | 5m | Critical |
 | [Workflow Task schedule-to-start latency severe](/troubleshooting/sdk-worker-capacity#workflow-task-schedule-to-start-latency-elevated) | `workflow_task_schedule_to_start_latency` | `task_queue` | p99 above 30m | 5m | Critical |
 | [Activity schedule-to-start latency severe](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated) | `activity_schedule_to_start_latency` | `task_queue` | p99 above 30m | 5m | Critical |
 | [Sticky cache holding zero entries under load](/troubleshooting/sdk-worker-capacity#sticky-cache-holding-zero-entries-under-load) | `sticky_cache_size` | Paired with a non-zero Workflow Task rate on the same Worker | Reaches 0 | 15m | Warning |
@@ -114,11 +114,12 @@ Triage guidance: [SDK execution failures](/troubleshooting/sdk-execution-failure
 
 | Failure mode | Metric | Condition | Threshold | `for` | Default severity |
 | --- | --- | --- | --- | --- | --- |
-| [Non-determinism error](/troubleshooting/sdk-execution-failures#non-determinism-error) | `workflow_task_execution_failed` | `failure_reason=NonDeterminismError` | Any occurrence | 5m | Critical |
+| [Non-determinism error](/troubleshooting/sdk-execution-failures#non-determinism-error) | `workflow_task_execution_failed` | `failure_reason=NonDeterminismError` | Any occurrence | 1m | Critical |
 | [gRPC message too large](/troubleshooting/sdk-execution-failures#grpc-message-too-large) | `workflow_task_execution_failed` | `failure_reason=GrpcMessageTooLarge` | Any occurrence | 1m | Critical |
 | [Workflow Task execution failures elevated](/troubleshooting/sdk-execution-failures#workflow-task-execution-failures-elevated) | `workflow_task_execution_failed` | `failure_reason=WorkflowError` | Rate above 10/s | 2m | Warning |
 | [Workflow Task execution latency high](/troubleshooting/sdk-execution-failures#workflow-task-execution-latency-high) | `workflow_task_execution_latency` | `task_queue`, `workflow_type` | p99 above 10s | 5m | Critical |
 | [Activity execution failures elevated](/troubleshooting/sdk-execution-failures#activity-execution-failures-elevated) | `activity_execution_failed` | `activity_type` | Rate above 10/s | 2m | Warning |
+| [Unregistered Activity invocation](/troubleshooting/sdk-execution-failures#unregistered-activity-invocation) — Go SDK only | `unregistered_activity_invocation` | `activity_type`, `task_queue`, `workflow_type` | Any occurrence | 1m | Critical |
 | [Local Activity latency exceeds the heartbeat timeout](/troubleshooting/sdk-execution-failures#local-activity-latency-exceeds-the-heartbeat-timeout) | `local_activity_execution_latency` | `activity_type` | p99 above 30m | 5m | Critical |
 
 ## Set up the alerts {/* #set-up-the-alerts */}
@@ -138,8 +139,8 @@ The names you query depend on your SDK and metrics reporter:
 - **Histograms** take a `_seconds_bucket` suffix on the underlying bucket series: `temporal_workflow_task_schedule_to_start_latency_seconds_bucket`.
 - **Gauges take no suffix.** `temporal_num_pollers`, `temporal_worker_task_slots_available`, and `temporal_sticky_cache_size` are gauges. Querying them with `_total` returns no data.
 
-Tag values also vary.
-The `status_code` tag is `PascalCase` in the Go SDK (`NotFound`) and `UPPER_SNAKE_CASE` in the Java and Core SDKs (`NOT_FOUND`).
+Tag coverage also varies.
+`status_code` values are `UPPER_SNAKE_CASE` in every SDK, matching the gRPC status code names (`NOT_FOUND`, `RESOURCE_EXHAUSTED`), and the tag can be suppressed through Client options.
 Some tags are not emitted by every SDK — `sticky_cache_size`, for example, carries `namespace` only in the TypeScript and Java SDKs and `task_queue` only in TypeScript.
 
 Confirm the exact names and tags in your own metrics endpoint before writing queries.

docs/references/sdk-metrics.mdx

@@ -64,9 +64,9 @@ Each metric may have some combination of the following [tags](/glossary#tag) att
 - `namespace`: Namespace the Worker is bound to
 - `poller_type`: One of the following:
   - `workflow_task`
+  - `workflow_sticky_task`
   - `activity_task`
   - `nexus_task`
-  - `sticky_workflow_task`
 - `worker_type`: One of the following:
   - `ActivityWorker`
   - `WorkflowWorker`
@@ -75,6 +75,8 @@ Each metric may have some combination of the following [tags](/glossary#tag) att
 - `activity_type`: The name of the Activity Function the metric is associated with
 - `workflow_type`: The name of the Workflow Function the metric is associated with
 - `operation`: RPC method name; available for metrics related to Temporal Client gRPC requests
+- `status_code`: gRPC status code returned by the Temporal Service, in `UPPER_SNAKE_CASE` (for example, `NOT_FOUND`); available on Temporal Client gRPC request failure metrics
+- `cause`: reason the Temporal Service rejected a request as resource exhausted (for example, `RESOURCE_EXHAUSTED_CAUSE_RPS_LIMIT`); available on the resource-exhausted metrics
 
 Some tags may not be available in every SDK, and Histogram metrics may have different buckets in each SDK.
 
@@ -96,6 +98,7 @@ Some tags may not be available in every SDK, and Histogram metrics may have diff
 | [temporal_long_request](#long_request)                                                       | Service Client | Counter     | Core, Go, Java |
 | [temporal_long_request_failure](#long_request_failure)                                       | Service Client | Counter     | Core, Go, Java |
 | [temporal_long_request_latency](#long_request_latency)                                       | Service Client | Histogram   | Core, Go, Java |
+| [temporal_long_request_resource_exhausted](#long_request_resource_exhausted)                 | Service Client | Counter     | Go             |
 | [temporal_nexus_poll_no_task](#nexus_poll_no_task)                                           | Worker         | Counter     | Core, Go, Java |
 | [temporal_nexus_task_schedule_to_start_latency](#nexus_task_schedule_to_start_latency)       | Worker         | Histogram   | Core, Go, Java |
 | [temporal_nexus_task_execution_failed](#nexus_task_execution_failed)                         | Worker         | Counter     | Core, Go, Java |
@@ -106,6 +109,7 @@ Some tags may not be available in every SDK, and Histogram metrics may have diff
 | [temporal_request](#request)                                                                 | Service Client | Counter     | Core, Go, Java |
 | [temporal_request_failure](#request_failure)                                                 | Service Client | Counter     | Core, Go, Java |
 | [temporal_request_latency](#request_latency)                                                 | Service Client | Histogram   | Core, Go, Java |
+| [temporal_request_resource_exhausted](#request_resource_exhausted)                           | Service Client | Counter     | Go             |
 | [temporal_resource_slots_cpu_usage](#resource_slots_cpu_usage)                               | Worker         | Gauge       | Core, Java     |
 | [temporal_resource_slots_mem_usage](#resource_slots_mem_usage)                               | Worker         | Gauge       | Core, Java     |
 | [temporal_sticky_cache_hit](#sticky_cache_hit)                                               | Worker         | Counter     | Core, Go, Java |
@@ -256,7 +260,7 @@ This number is included into the total `long_request` counter for long poll RPC
 - Tags: `namespace`, `operation`, `status_code`
 
 The `status_code` tag carries the gRPC status code returned by the Temporal Service.
-Its casing differs by SDK: the Go SDK reports `PascalCase` values such as `NotFound`, while the Java and Core SDKs report `UPPER_SNAKE_CASE` values such as `NOT_FOUND`.
+Values are `UPPER_SNAKE_CASE`, matching the gRPC status code names: `NOT_FOUND`, `RESOURCE_EXHAUSTED`, `UNIMPLEMENTED`, `INTERNAL`, and so on.
 The tag can be suppressed through Client options, so confirm it is present in your own metrics endpoint before writing queries against it.
 
 ### `long_request_latency`
@@ -267,6 +271,18 @@ Latency of a Temporal Client gRPC long poll request.
 - Available in: Core, Go, Java
 - Tags: `namespace`, `operation`
 
+### `long_request_resource_exhausted`
+
+Temporal Client made an RPC long poll request that the Temporal Service rejected as resource exhausted.
+These failures are also counted in `long_request_failure` with `status_code=RESOURCE_EXHAUSTED`; this metric adds the reason for the rejection.
+
+- Type: Counter
+- Available in: Go
+- Tags: `namespace`, `operation`, `cause`
+
+The `cause` tag carries the reason the Temporal Service rejected the request, such as `RESOURCE_EXHAUSTED_CAUSE_RPS_LIMIT`, `RESOURCE_EXHAUSTED_CAUSE_CONCURRENT_LIMIT`, `RESOURCE_EXHAUSTED_CAUSE_SYSTEM_OVERLOADED`, or `RESOURCE_EXHAUSTED_CAUSE_CIRCUIT_BREAKER_OPEN`.
+The cause determines the remedy, so prefer this metric over `long_request_failure` when investigating throttling.
+
 ### `nexus_poll_no_task`
 
 A Nexus Worker poll for a Nexus Task timed out, and no Nexus Task is available to pick from the Task Queue.
@@ -359,7 +375,7 @@ This number is included into the total `request` counter for RPC requests.
 - Tags: `namespace`, `operation`, `status_code`
 
 The `status_code` tag carries the gRPC status code returned by the Temporal Service.
-Its casing differs by SDK: the Go SDK reports `PascalCase` values such as `NotFound`, while the Java and Core SDKs report `UPPER_SNAKE_CASE` values such as `NOT_FOUND`.
+Values are `UPPER_SNAKE_CASE`, matching the gRPC status code names: `NOT_FOUND`, `RESOURCE_EXHAUSTED`, `UNIMPLEMENTED`, `INTERNAL`, and so on.
 The tag can be suppressed through Client options, so confirm it is present in your own metrics endpoint before writing queries against it.
 
 ### `request_latency`
@@ -370,6 +386,18 @@ Latency of a Temporal Client gRPC request.
 - Available in: Core, Go, Java
 - Tags: `namespace`, `operation`
 
+### `request_resource_exhausted`
+
+Temporal Client made an RPC request that the Temporal Service rejected as resource exhausted.
+These failures are also counted in `request_failure` with `status_code=RESOURCE_EXHAUSTED`; this metric adds the reason for the rejection.
+
+- Type: Counter
+- Available in: Go
+- Tags: `namespace`, `operation`, `cause`
+
+The `cause` tag carries the reason the Temporal Service rejected the request, such as `RESOURCE_EXHAUSTED_CAUSE_RPS_LIMIT`, `RESOURCE_EXHAUSTED_CAUSE_CONCURRENT_LIMIT`, `RESOURCE_EXHAUSTED_CAUSE_SYSTEM_OVERLOADED`, or `RESOURCE_EXHAUSTED_CAUSE_CIRCUIT_BREAKER_OPEN`.
+The cause determines the remedy, so prefer this metric over `request_failure` when investigating throttling.
+
 ### `resource_slots_cpu_usage`
 
 CPU usage as a value between 0 and 100. As perceived by the resource-based slots tuner, if

docs/troubleshooting/sdk-execution-failures.mdx

@@ -44,7 +44,7 @@ A non-determinism error does not resolve on its own.
 1. **Read the error.** The `WorkflowTaskFailed` event in an affected Execution's Event History contains the message identifying exactly where replay diverged and which command was expected versus produced. This is the most direct signal for root cause.
 1. **Determine whether this is a code change or a deploy artifact.** Common causes:
    - A code change added, removed, or reordered commands — Activity scheduling, Timers, Signals, Child Workflows — without a versioning guard. In-flight Executions that built History under the old code fail on the new code.
-   - A rolling restart with old and new Worker versions briefly running together. Some Executions fail transiently and recover once the rollout completes, which is why this alert needs a `for` duration longer than your deploy takes.
+   - A rolling restart with old and new Worker versions briefly running together. Some Executions fail transiently and recover once the rollout completes. If your deploys routinely trigger this alert, lengthen its `for` duration past how long a rollout takes.
    - Changed Activity or Timer parameters in existing Workflow code without versioning.
 1. **Roll back if it is not resolving.** If the errors started after a deploy and are not clearing on their own, roll the Worker back to the previous version. Affected Executions resume on their next Workflow Task retry once compatible code is running. Then introduce a proper versioning guard before redeploying — see [Versioning Workflows](/workflow-definition#workflow-versioning) and [Worker Versioning](/production-deployment/worker-deployments/worker-versioning).
 1. **Watch Worker pressure.** Continuous retries put sustained load on Workflow Workers. Cross-check [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) for `worker_type=WorkflowWorker` and [Workflow Task execution latency high](#workflow-task-execution-latency-high) — a high volume of retries can saturate capacity and affect healthy Executions on the same Task Queue.
@@ -152,7 +152,29 @@ At scale, sustained retry bursts put significant pressure on Task matching and t
 1. **Determine whether this is transient or a bug.** A downstream service outage, network partition, or database timeout recovers on its own — watch whether the rate falls. A persistent code bug does not.
 1. **Check downstream service health.** A degraded dependency is a common cause of sustained failure bursts. If the dependency is throttling, confirm your [Retry Policy](/encyclopedia/retry-policies) has appropriate backoff — without it, retry bursts amplify the pressure you are already applying.
 1. **Check schedule-to-start latency.** A growing retry backlog shows up as elevated [Activity schedule-to-start latency](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated) even after the failure rate drops.
-1. **Mark expected failures as benign.** If your design intentionally fails Activities — polling patterns, Saga compensations, flow control through exceptions — mark those `ApplicationFailure` instances with category `BENIGN`. All SDKs support this and suppress this metric for them, which lets this alert track unexpected failures without per-`activity_type` threshold tuning.
+1. **Mark expected failures as benign.** If your design intentionally fails Activities — polling patterns, Saga compensations, flow control through exceptions — mark those `ApplicationFailure` instances with category `BENIGN`, which suppresses this metric for them and lets the alert track unexpected failures without per-`activity_type` threshold tuning. Confirm your SDK version supports the category before relying on it.
+
+Note that internal failures — a context propagation error or a context timeout, rather than an Activity returning a failure — increment this counter regardless of category.
+
+## Unregistered Activity invocation {/* #unregistered-activity-invocation */}
+
+**Metric:** [`temporal_unregistered_activity_invocation`](/references/sdk-metrics#unregistered_activity_invocation), tagged `activity_type`, `task_queue`, and `workflow_type`
+
+A Workflow scheduled an Activity that the Worker polling that Task Queue has no registered implementation for.
+
+This metric is emitted by the Go SDK only.
+
+**Why it matters.**
+The Activity cannot execute. It will keep being retried against a Worker that cannot run it until the Activity's `scheduleToClose` timeout expires, or indefinitely if no such timeout is set — so the Workflow Execution waiting on it makes no progress.
+
+This is almost always a deployment error rather than a runtime condition: Workflow code that schedules an Activity the deployed Worker does not register. It does not resolve on its own.
+
+**Triage.**
+
+1. **Identify the Activity and the Task Queue.** The `activity_type` and `task_queue` tags name both. The `workflow_type` tag identifies which Workflow is scheduling it.
+1. **Check whether the Activity is registered on the right Worker.** Confirm the Worker polling that Task Queue registers that Activity type. A common cause is registering the Activity on a Worker polling a different Task Queue.
+1. **Check for a partial rollout.** If Workflow code that schedules a new Activity deployed ahead of the Worker that implements it, some Workers will be running without the registration. Complete the rollout.
+1. **Check for a renamed Activity.** Changing an Activity's registered name while Executions are in flight leaves those Executions scheduling the old name. Register both names until the in-flight Executions drain, or use a versioning guard.
 
 ## Local Activity latency exceeds the heartbeat timeout {/* #local-activity-latency-exceeds-the-heartbeat-timeout */}