🐛 Bug Report: [anthropic] Streaming spans drop gen_ai.response.finish_reasons and gen_ai.output.messages for turns with empty content

[dynomisto]

Which component is this bug for?

Anthropic Instrumentation

📜 Description

Package: opentelemetry-instrumentation-anthropic
Version: 0.62.1

Summary

When a streamed Anthropic response ends with a valid stop_reason (e.g. end_turn) but an empty content array — the model returns a turn with no content blocks at all — the generated anthropic.chat span records neither gen_ai.response.finish_reasons nor gen_ai.output.messages, even though the API did return a stop_reason.

Non-streaming responses and streaming responses that contain at least one content block are recorded correctly, so this is specific to the streaming path when the content array is empty.

Root cause

In opentelemetry/instrumentation/anthropic/streaming.py, _process_response_item handles the message_delta chunk (which is where the message-level stop_reason arrives) by writing the stop reason only onto already-accumulated per-content-block events:

elif item.type == "message_delta":
    for event in complete_response.get("events", []):
        event["finish_reason"] = item.delta.stop_reason
    ...

When the response has no content blocks, no content_block_start items are ever received, so complete_response["events"] stays empty and this loop is a no-op — item.delta.stop_reason is discarded. The stop reason is never stored at the message level on complete_response.

set_streaming_response_attributes in span_utils.py then receives that empty events list and returns immediately:

def set_streaming_response_attributes(span, complete_response_events):
    if not span.is_recording() or not complete_response_events:
        return
    ...

so neither gen_ai.response.finish_reasons nor gen_ai.output.messages is set.

By contrast, the non-streaming path (_set_span_completions in span_utils.py) reads response.stop_reason directly and always records finish_reasons when a stop reason is present, which is why only the streaming/empty-content case is affected.

👟 Reproduction steps

Reproduction (deterministic, no network / credentials)

This drives the instrumentation's own stream-accumulation functions with synthetic stream items, so it isolates the bug without an API call:

from types import SimpleNamespace

from opentelemetry.instrumentation.anthropic import span_utils, streaming
from opentelemetry.semconv._incubating.attributes import gen_ai_attributes as GA


class FakeSpan:
    def __init__(self):
        self.attrs = {}

    def is_recording(self):
        return True

    def set_attribute(self, key, value):
        self.attrs[key] = value


def accumulate(items):
    complete_response = {"events": [], "model": "", "usage": {}, "id": ""}
    for item in items:
        streaming._process_response_item(item, complete_response)
    span = FakeSpan()
    span_utils.set_streaming_response_attributes(span, complete_response.get("events"))
    return span.attrs


message_delta = SimpleNamespace(
    type="message_delta",
    delta=SimpleNamespace(stop_reason="end_turn"),
    usage={"output_tokens": 4},
)

# Empty-content turn: message_delta carries stop_reason="end_turn" but no content blocks.
print("empty content   :", accumulate([message_delta]))

# Content-bearing turn, for contrast.
block_start = SimpleNamespace(
    type="content_block_start", index=0,
    content_block=SimpleNamespace(type="text"),
)
block_delta = SimpleNamespace(
    type="content_block_delta", index=0,
    delta=SimpleNamespace(type="text_delta", text="hi"),
)
print("with content    :", accumulate([block_start, block_delta, message_delta]))

👍 Expected behavior

Expected

The empty-content turn should still record gen_ai.response.finish_reasons (e.g. ['stop']), matching the non-streaming path. gen_ai.output.messages may reasonably be omitted or emitted as an empty assistant message, but the finish reason should not be lost.

👎 Actual Behavior with Screenshots

Observed

empty content   : {}
with content    : {'gen_ai.response.finish_reasons': ['stop'], 'gen_ai.output.messages': '[...]'}

For the empty-content turn, no attributes are set at all — finish_reasons is dropped despite stop_reason="end_turn" being present on the message_delta.

🤖 Python Version

3.12

📃 Provide any additional context for the Bug.

How this happens in practice

The empty-content turn is a normal Anthropic API behavior: the model returns a message with content == [] and stop_reason == "end_turn" when it has nothing further to emit — for example a final turn that follows a tool result and adds no additional text. That turn is fully valid; it simply carries no content blocks, which is enough to trigger the drop.

Suggested fix

In _process_response_item, also record the message-level stop reason on complete_response itself when handling message_delta (independent of whether any content-block events exist), e.g. complete_response["stop_reason"] = item.delta.stop_reason. Then in set_streaming_response_attributes, set gen_ai.response.finish_reasons from that message-level stop reason as a fallback when the per-event finish reasons are empty (and avoid the early return when there are no content events but a stop reason is present).

Environment

• opentelemetry-instrumentation-anthropic 0.62.1
• Anthropic Python SDK, streaming (messages.stream(...))
• Python 3.12

👀 Have you spent some time to check if this bug has been raised before?

• I checked and didn't find similar issue

Are you willing to submit PR?

None

[Solaris-star]

Opened a fix in #4372 — store message-level stop_reason on empty-content streams and use it as a finish_reasons fallback.