feat(telemetry): record tool definitions on the llm_node span - #6621
Open
biztex wants to merge 2 commits into
Open
feat(telemetry): record tool definitions on the llm_node span#6621biztex wants to merge 2 commits into
biztex wants to merge 2 commits into
Conversation
The llm_node span records the full chat context but only the names of function tools (lk.function_tools). Tool descriptions and parameter schemas never reach OTel backends, so a trace cannot show what the LLM was actually offered on a turn - a badly-worded tool description or a schema mismatch is invisible when tracing back a conversation that went wrong. Add lk.function_tool_definitions: a JSON attribute with each tool's name, description and parameters schema - the same shape providers receive (build_legacy_openai_schema for function tools, raw_schema for raw tools). lk.function_tools stays unchanged for backward compatibility, and a tool whose schema fails to build falls back to name-only so telemetry can never break generation. Fixes livekit#6620
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
The per-tool guard covered schema building but json.dumps ran outside it, so a tool definition carrying a non-JSON value could abort the model turn for the sake of optional debugging data. Serialize with default=str and fall back to the name list if serialization still fails; the span attribute degrades, generation never breaks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6620
Problem
The
llm_nodespan records the full chat context (lk.chat_ctx) but only the names of function tools (lk.function_tools). Tool descriptions and parameter schemas never reach OTel backends (Langfuse, Jaeger, ...), so a trace cannot show what the LLM was actually offered on a turn — a badly-worded tool description or a schema mismatch is invisible when tracing back a conversation that went wrong. Reported by a user in the community Slack running Langfuse viaset_tracer_provider.Change
Adds
lk.function_tool_definitionsto thellm_nodespan: a JSON attribute carrying each tool's name, description, and parameters schema — the same shape providers receive (build_legacy_openai_schema(..., internally_tagged=True)for function tools,raw_schemafor raw tools).lk.function_tools(names) is unchanged, for backward compatibility with anything keying on it.Tests
New hermetic
tests/test_llm_span_tool_definitions.py(unit-marked): definition shape for decorated and raw tools, JSON round-trip, and an end-to-end span assertion drivingperform_llm_inferencewith a fake LLM node against an in-memory span exporter (the global tracer provider is restored afterwards).