- Author(s): Yash Tibrewal (@yashykt)
- Approver: Mark Roth (@markdroth), Eric Anderson (@ejona86), Doug Fawley (@dfawley)
- Status: Ready for Implementation
- Implemented in:
- Last updated: 2025-12-08
- Discussion at: https://groups.google.com/g/grpc-io/c/iMdK7r4E5tU
Introduce OpenTelemetry metrics for subchannels. These metrics will replace the existing pick-first metrics.
In A78, metrics for PickFirst load-balancing policy were proposed that provide observability on disconnections for subchannels and connection attempts made for those subchannels. These metrics do not currently contain information on the reason for disconnection, the xds locality or the cluster information.
A89 is a proposal to introduce a new optional label grpc.lb.backend_service
to client-side per-attempt metrics. This label has xds cluster information.
- A8: Client-side Keepalive
- A18: TCP User Timeout
- A61: IPv4 and IPv6 Dualstack Backend Support
- A66: OpenTelemetry Metrics
- A74: xDS Config Tears
- A78: gRPC OTel Metrics for WRR, Pick First, and XdsClient
- A79: Non-per-call Metrics Architecture
- A89: Backend Service Metric Label
- L62: gRPC security level negotiation between call credentials and channels
Move the existing pick-first metrics to subchannel metrics
(grpc.lb.pick_first.* to grpc.subchannel.*) with the addition of optional
labels as shown below -
| Metric Name | Type | Unit | Labels | Description |
|---|---|---|---|---|
| grpc.subchannel.disconnections (Old - grpc.lb.pick_first.disconnections) | Counter | {disconnection} | grpc.target, grpc.lb.backend_service (optional), grpc.lb.locality (optional), grpc.disconnect_error (optional) | Number of times the selected subchannel becomes disconnected. |
| grpc.subchannel.connection_attempts_succeeded (Old - grpc.lb.pick_first.connection_attempts_succeeded) | Counter | {attempt} | grpc.target, grpc.lb.backend_service (optional), grpc.lb.locality (optional) | Number of successful connection attempts. |
| grpc.subchannel.connection_attempts_failed (Old - grpc.lb.pick_first.connection_attempts_failed) | Counter | {attempt} | grpc.target, grpc.lb.backend_service (optional), grpc.lb.locality (optional) | Number of failed connection attempts. |
| grpc.subchannel.open_connections | UpDown Counter | {connection} | grpc.target, grpc.security_level (optional), grpc.lb.backend_service (optional), grpc.lb.locality (optional) | Number of open connections. |
If the subchannel is shut down while a connection attempt is in flight, we should not record the connection attempt as either succeeded or failed.
Implementations that have already implemented the pick-first metrics should give enough time for users to transition to the new metrics. For example, implementations should report both the old pick-first metrics and the new subchannel metrics for 2 releases, and then remove the old pick-first metrics.
| Label Name | Disposition | Description |
|---|---|---|
| grpc.target | Required | Indicates the target of the gRPC channel (defined in A66.) |
| grpc.lb.backend_service | Optional | The backend service to which the RPC was routed (defined in A89.) |
| grpc.lb.locality | Optional | The locality to which the traffic is being sent (defined in A78 -- but see below for differences.) |
| grpc.disconnect_error | Optional | Reason for disconnection. |
| grpc.security_level | Optional | Denotes the security level of the connection. Allowed values - "none", "integrity_only" and "privacy_and_integrity". |
The subchannel needs to be passed attributes with the values for the
grpc.lb.backend_service and grpc.lb.locality labels (defined in A89 and
A78 respectively). This implies that the subchannel will be recreated when
these attributes change. Since currently, only xDS is using these labels, the
attributes will be set for each endpoint or address by cds (post-A74) or
xds_cluster_resolver (pre-A74) LB policies.
Note that for the grpc.lb.locality label, we cannot simply reuse the
top-level attribute passed down from the weighted_target policy to the WRR
policy, as described in A78. The WRR policy needs a top-level attribute
rather than a per-endpoint attribute, since it needs a single value
to use for all endpoints. However, in Java and Go, only per-endpoint
attributes are passed to the subchannel, so the subchannel cannot use
the top-level attribute. Furthermore, even in C-core, where top-level
attributes are passed to the subchannel, we are populating the attribute
at a point in the LB policy tree where we have endpoints for multiple
localities, which necessitates the use of a per-endpoint attribute.
As a result, although the WRR policy will continue to use the top-level
attribute set by the weighted_target policy, we will need to create a
new per-endpoint attribute with the same value for use by the subchannel.
Similarly, for the grpc.lb.backend_service label, Java and Go will not
be able to simply reuse the existing top-level attribute passed down from
the cds (post-[A75]) or xds_cluster_impl (pre-[A75]) policies to the WRR
policy, as described in A89, since that top-level attribute will not be
passed to the subchannel. Therefore, those implementations will need
to create a duplicate of this attribute as a per-endpoint attribute.
However, for this label, the attribute will apply to all endpoints,
so C-core can continue to set it as a top-level attribute.
List of allowed values for grpc.disconnect_error -
| Error string | Description |
|---|---|
| GOAWAY <ERROR_CODE> | HTTP2 GOAWAY frame with error code for example (“GOAWAY NO_ERROR”, “GOAWAY PROTOCOL_ERROR”, “GOAWAY ENHANCE_YOUR_CALM”). The list of error codes is available in RFC 9113. |
| subchannel shutdown | The subchannel was shutdown. This can happen due to reasons such as the parent channel shutting down, channel becoming idle, the load balancing policy changing due to a resolver update, or a change in list of endpoint addresses. |
| connection reset | Connection was reset (eg. ECONNRESET, WSAECONNERESET.) |
| connection timed out | Connection timed out (eg. ETIMEDOUT, WSAETIMEDOUT), also includes connections closed due to A8: gRPC keepalives. |
| connection aborted | Connection was aborted (eg. ECONNABORTED, WSAECONNABORTED.) |
| socket error | Any socket error not covered by “connection reset”, “connection timed out” and “connection aborted”. Implementations that are not able to differentiate between the different socket error codes should also use this. |
| unknown | Catch-all for all other reasons. |
For a given connection, there can be multiple reasons reported to the subchannel
for disconnection. For example, a connection could have seen a GOAWAY frame with
ENHANCE_YOUR_CALM and then a socket error Broken Pipe. In such cases, the
first seen reason should be chosen, GOAWAY ENHANCE_YOUR_CALM in this case.
We might add more error cases to this in the future.
As recommended by A79, these metrics will start off as experimental, and hence off-by-default. The decision on whether these metrics will be on-by-default or off-by-default on de-experimentalization will be made at the same time as the de-experimentalization.
The existing pick-first metrics provides stats on subchannel disconnections and connection attempts as viewed from the perspective of the pick-first lb policy. A61 made pick-first lb policy the universal leaf policy. For users unfamiliar with this, it will come as a surprise when metrics for pick-first lb policy are populated when round_robin lb policy is configured (for example). Additionally, the pick-first metrics are defined from the perspective of the channel. This means that if subchannels are shared between multiple channels (as is the case for gRPC Core and its wrapped languages - C++, Python), we will double-count the disconnections/connection attempts.
Renaming/moving the pick-first metrics to subchannel makes this more intuitive, and fixes the double-counting problem.
Moving the metrics down to subchannel potentially allows us to calculate the
number of open connections by subtracting grpc.subchannel.disconnections from
grpc.subchannel.connection_attempts_succeeded. This method does not work for
exporters recording counters per period in a way that does not allow for a
simple subtraction of the two counters
(grpc/grpc#34886).
Adding an explicit metric that records the number of open connections avoids this.
We expect most implementations of A8 to also set the POSIX socket option
TCP_USER_TIMEOUT with the same timeout value as stated in A18. As such, in
cases where the connection is broken, the keepalive timeout will race with
sockets being closed due to TCP_USER_TIMEOUT. Since the motive of the two
timers is essentially the same, we choose to combine them into a single error,
instead of trying to differentiate between them.
TBD