GenAI Utils | Inference Type and Span Creation by keith-decker · Pull Request #3768 · open-telemetry/opentelemetry-python-contrib

keith-decker · 2025-09-18T14:54:54Z

Description

The GenAI Utils package will include boilerplate and helpers to standardize instrumentation for Generative AI.
This PR provides APIs to minimize the work needed to instrument genai libraries. It provides helpers for starting and stopping LLM invocations, it generates spans aligned with semconvs.

Metrics and Events to come on future PRs

Type of change

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Basic unit test for starting and stopping LLM - Spans
Unit test for parent/child invocations

Does This PR Require a Core Repo Change?

Yes. - Link to PR:
No.

Checklist:

See contributing.md for styleguide, changelog guidelines, and more.

Followed the style guidelines of this project
Changelogs have been updated
Unit tests have been added
Documentation has been updated

keith-decker · 2025-09-18T14:55:48Z

A cleaned up version of: #3732

Flattened LLMInvocation to not use dictionary keys
Removed the parent run_id, use OTEL context to manage parent/child relationships on spans
Return the run_id on creation if one is not provided, in order to specify which span to end on finish

aabmass · 2025-09-18T16:27:06Z

@keith-decker thing I was mentioning in the SIG:
Looks like you already figured out the context manager thing but figured I'd share what I wrote:

opentelemetry-python-contrib/instrumentation-genai/opentelemetry-instrumentation-vertexai/src/opentelemetry/instrumentation/vertexai/patch.py

Lines 101 to 107 in ee947ce

    
           def _with_instrumentation( 
        
               self, 
        
               instance: client.PredictionServiceClient 
        
               | client_v1beta1.PredictionServiceClient, 
        
               args: Any, 
        
               kwargs: Any, 
        
           ):

It's used for both sync and async wrapper, and would work for streaming as well.

util/opentelemetry-util-genai/src/opentelemetry/util/genai/handler.py

util/opentelemetry-util-genai/src/opentelemetry/util/genai/span_utils.py

…anGenerator

util/opentelemetry-util-genai/src/opentelemetry/util/genai/span_utils.py

util/opentelemetry-util-genai/src/opentelemetry/util/genai/types.py

util/opentelemetry-util-genai/src/opentelemetry/util/genai/handler.py

…ssages for clarity

util/opentelemetry-util-genai/src/opentelemetry/util/genai/span_utils.py

…agement to LLMInvocation class

util/opentelemetry-util-genai/src/opentelemetry/util/genai/handler.py

util/opentelemetry-util-genai/src/opentelemetry/util/genai/generators.py

…_llm

DylanRussell · 2025-09-30T14:15:26Z

This looks good to me now. Just added a few comments

util/opentelemetry-util-genai/src/opentelemetry/util/genai/handler.py

keith-decker · 2025-10-08T16:29:14Z

@aabmass genai-file upload on python 3.9 seems to be flakey. I've seen it fail occasionally, and I don't have the ability to restart the test.

aabmass · 2025-10-09T16:02:56Z

Sorry about that tracking here #3812 (no idea why)

…r setup

aabmass

Mostly nits, thanks for your patience!

util/opentelemetry-util-genai/src/opentelemetry/util/genai/types.py

util/opentelemetry-util-genai/src/opentelemetry/util/genai/handler.py

util/opentelemetry-util-genai/src/opentelemetry/util/genai/span_utils.py

util/opentelemetry-util-genai/src/opentelemetry/util/genai/handler.py

util/opentelemetry-util-genai/src/opentelemetry/util/genai/span_utils.py

util/opentelemetry-util-genai/src/opentelemetry/util/genai/handler.py

…lization

…xt manually" This reverts commit be8620b.

* cherry pick changes from previous PR * move span utils to new file * remove span state, use otel context for parent/child * flatten LLMInvocation to use attributes instead of dict keys * helper function and docstrings * refactor: store span and context token in LLMInvocation instead of SpanGenerator * refactor: rename prompts/chat_generations to input_messages/output_messages for clarity * refactor: simplify TelemetryHandler API by moving invocation data management to LLMInvocation class * refactor: update relative imports to absolute imports * Update handler to use a context manager instead of start_llm and stop_llm * resolve tox -e doc failure * safeguard against empty request-model * fix tox typecheck errors for utils * refactor: move tracer to generator, clean up dead code * remove unused linting hint * back off stricter request-model requirements * reintroduce manual start/stop for langchain callback flow * clean up context handler, clarify unit tests * remove generator concept * update token types * code cleanup * Refactor TestTelemetryHandler to use instance method for span exporter setup * refactor: remove unused type properties * refactor: update TelemetryHandler initialization to remove **kwargs * refactor: remove tracer variable * refactor: code style updates * refactor: replace json.dumps with gen_ai_json_dumps for message serialization * refactor: update span lifecycle to use sdk over setting context manually * refactor: don't reinvent span attribute assignment * refactor: pylint update for python 3.13 * Revert "refactor: update span lifecycle to use sdk over setting context manually" This reverts commit be8620b. --------- Co-authored-by: Aaron Abbott <aaronabbott@google.com>

keith-decker added 5 commits September 17, 2025 09:12

cherry pick changes from previous PR

73a764c

move span utils to new file

0d749a9

remove span state, use otel context for parent/child

77d9c3c

flatten LLMInvocation to use attributes instead of dict keys

054ebe9

helper function and docstrings

9d3926f

keith-decker requested a review from a team as a code owner September 18, 2025 14:54

github-actions bot assigned DylanRussell and keith-decker Sep 18, 2025

github-actions bot requested a review from DylanRussell September 18, 2025 14:55

keith-decker mentioned this pull request Sep 18, 2025

GenAI Utils | Initial Helpers for Inference #3732

Closed

9 tasks

Merge branch 'main' into util-genai-inference-clean

635b7f8

DylanRussell reviewed Sep 18, 2025

View reviewed changes

refactor: store span and context token in LLMInvocation instead of Sp…

9837cf4

…anGenerator

DylanRussell reviewed Sep 19, 2025

View reviewed changes

util/opentelemetry-util-genai/src/opentelemetry/util/genai/span_utils.py Outdated Show resolved Hide resolved

DylanRussell reviewed Sep 19, 2025

View reviewed changes

util/opentelemetry-util-genai/src/opentelemetry/util/genai/types.py Outdated Show resolved Hide resolved

DylanRussell reviewed Sep 19, 2025

View reviewed changes

util/opentelemetry-util-genai/src/opentelemetry/util/genai/handler.py Outdated Show resolved Hide resolved

refactor: rename prompts/chat_generations to input_messages/output_me…

1a172d1

…ssages for clarity

DylanRussell reviewed Sep 19, 2025

View reviewed changes

util/opentelemetry-util-genai/src/opentelemetry/util/genai/span_utils.py Show resolved Hide resolved

DylanRussell reviewed Sep 19, 2025

View reviewed changes

util/opentelemetry-util-genai/src/opentelemetry/util/genai/span_utils.py Outdated Show resolved Hide resolved

keith-decker added 2 commits September 19, 2025 11:03

refactor: simplify TelemetryHandler API by moving invocation data man…

465ca78

…agement to LLMInvocation class

refactor: update relative imports to absolute imports

cd5aaa6

DylanRussell reviewed Sep 22, 2025

View reviewed changes

util/opentelemetry-util-genai/src/opentelemetry/util/genai/handler.py Show resolved Hide resolved

DylanRussell approved these changes Sep 22, 2025

View reviewed changes

DylanRussell reviewed Sep 22, 2025

View reviewed changes

util/opentelemetry-util-genai/src/opentelemetry/util/genai/generators.py Outdated Show resolved Hide resolved

keith-decker added 4 commits September 22, 2025 09:32

Update handler to use a context manager instead of start_llm and stop…

8347d17

…_llm

resolve tox -e doc failure

742a36f

safeguard against empty request-model

00e08a8

Merge branch 'main' into util-genai-inference-clean

109625c

keith-decker added 2 commits September 30, 2025 09:45

code cleanup

51d28c0

Merge branch 'main' into util-genai-inference-clean

88d0149

pmcollins reviewed Oct 1, 2025

View reviewed changes

util/opentelemetry-util-genai/src/opentelemetry/util/genai/handler.py Show resolved Hide resolved

Merge branch 'main' into util-genai-inference-clean

188e84b

nagkumar91 approved these changes Oct 2, 2025

View reviewed changes

keith-decker requested a review from aabmass October 3, 2025 16:54

keith-decker and others added 3 commits October 6, 2025 13:07

Merge branch 'main' into util-genai-inference-clean

c264c46

Merge branch 'main' into util-genai-inference-clean

61746ab

Merge branch 'main' into util-genai-inference-clean

d6001f5

Merge branch 'main' into util-genai-inference-clean

b96eb20

Refactor TestTelemetryHandler to use instance method for span exporte…

b1b2364

…r setup

aabmass approved these changes Oct 9, 2025

View reviewed changes

keith-decker and others added 10 commits October 9, 2025 11:47

refactor: remove unused type properties

51f0c8c

refactor: update TelemetryHandler initialization to remove **kwargs

7732024

refactor: remove tracer variable

839eeae

refactor: code style updates

4044274

refactor: replace json.dumps with gen_ai_json_dumps for message seria…

4280eb9

…lization

refactor: update span lifecycle to use sdk over setting context manually

be8620b

refactor: don't reinvent span attribute assignment

04d948a

refactor: pylint update for python 3.13

0d49e28

Revert "refactor: update span lifecycle to use sdk over setting conte…

3a891b2

…xt manually" This reverts commit be8620b.

Merge branch 'main' into util-genai-inference-clean

56ba760

aabmass merged commit e3d3817 into open-telemetry:main Oct 10, 2025
638 checks passed

wrisa mentioned this pull request Oct 20, 2025

Switched langchain llm callbacks to use genai utils handler #3889

Merged

11 tasks

nagkumar91 mentioned this pull request Mar 3, 2026

REQUEST: New membership for nagkumar91 open-telemetry/community#3304

Closed

6 tasks

keith-decker mentioned this pull request Mar 3, 2026

REQUEST: New membership for <keith-decker> open-telemetry/community#3305

Closed

6 tasks

Conversation

keith-decker commented Sep 18, 2025

Description

Type of change

How Has This Been Tested?

Does This PR Require a Core Repo Change?

Checklist:

Uh oh!

keith-decker commented Sep 18, 2025

Uh oh!

aabmass commented Sep 18, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DylanRussell commented Sep 30, 2025

Uh oh!

Uh oh!

keith-decker commented Oct 8, 2025

Uh oh!

aabmass commented Oct 9, 2025

Uh oh!

aabmass left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants