[Feature]: otelgrpc — capture `server.address` and `server.port` for failed RPC client spans

[cyrille-leclerc]

Component

Instrumentation: otelgrpc

Problem Statement

RPC client spans generated for gRPC calls that fail due to an invalid hostname are missing the server.address and server.port attributes.

These attributes are:

• Essential for effective troubleshooting
• Defined as “Conditionally Required If available” in the OpenTelemetry Semantic Conventions (reference)

See problem on the OTel Demo when activating the paymentUnreachable feature flag ( http://localhost:8080/feature ) as preported by @bpschmitt in open-telemetry/opentelemetry-demo#3203:

Proposed Solution

Add server.address and server.port for failed RPC client spans.

The source code seems to relate to the error handling part of:

opentelemetry-go-contrib/instrumentation/google.golang.org/grpc/otelgrpc/stats_handler.go

Lines 355 to 357 in 468d883

	if rs.Error != nil {
	s, _ = status.FromError(rs.Error)
	rpcStatusAttr = semconv.RPCResponseStatusCode(canonicalString(s.Code()))

Alternatives

No response

Prior Art

No response

Additional Context

No response

[dmathieu]

cc @dashpole as code owner.

[dashpole]

You might just need to update to the latest release. I just added #8723 recently, and it was released 5 days ago.

[cyrille-leclerc]

Thanks @dashpole , unfortunately, server.address and server.port are also missing with v0.68.0:

[dashpole]

Thats really interesting. Can you see if it exists on a non-error span? Just to see if it is related to the name resolver error.

[cyrille-leclerc]

We properly get server.address and server.port when gRPC calls are successful. The problem is specific to failed gRPC calls.

[dashpole]

It looks like when the request fails before headers are sent/received, we don't get an OutHeader event. Looking at the grpc-go implementation, they have both a stats.Handler and an interceptor. The interceptor adds some metadata to the context: https://github.com/grpc/grpc-go/blob/74b3acd1a801570e1cefb28cf61620a4ef7c8ee2/stats/opentelemetry/client_metrics.go#L72. The stats.Handler then extracts the metadata from the context.

But this would be a big change (requiring an interceptor). But I think we will have to go that route to be compliant with the semantic conventions.

[cyrille-leclerc]

Thanks for your thorough investigation.

[cyrille-leclerc]

@dashpole we observed in the OTel Demo traces that otelgrpc sets server.address to the IP address (e.g. 172.18.0.19) rather than the hostname provided to the gRPC client (e.g. payment).

This makes troubleshooting harder and seems misaligned with the OTel semantic conventions, which state that server.address should use the server domain name when available (without reverse DNS), falling back to the IP otherwise (see reference).

Would it make sense to open an enhancement request to address this?

[dashpole]

@cyrille-leclerc sure, that would be great. Is that a regression, or just something we should do?

[cyrille-leclerc]

Thanks David — I don’t think otelgrpc returning the IP address instead of the domain name for server.address is a regression; as far as I know, it’s always behaved that way.
It's great if we can change this.

FYI I created a dedicated enhancement request:

• [Feature]: otelgrpc: use hostname instead of the IP resolved from it for server.address #8789