GenAI spec updates for langchain#3813
GenAI spec updates for langchain#3813nagkumar91 wants to merge 37 commits intoopen-telemetry:mainfrom
Conversation
|
@wrisa, can you review it and propose the plan to implement in the opentelemetry-util-genai instead? |
@nagkumar91 we have latest WIP genai utils where all the semantic convention aligned telemetry can be moved. |
...ry-instrumentation-langchain/src/opentelemetry/instrumentation/langchain/callback_handler.py
Outdated
Show resolved
Hide resolved
|
I believe the ask here is to move common parts of content capturing (and anything else common) to https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/util/opentelemetry-util-genai |
|
This PR has been automatically marked as stale because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 days of this comment. |
|
@nagkumar91 What's more, do you have any plan to instrument the agent/tool/embedding process in langchain? We already implement it here and are glad to collaborate for this PR. |
MikeGoldsmith
left a comment
MikeGoldsmith
left a comment
There was a problem hiding this comment.
Thanks @nagkumar91. The spec alignment and capture_messages opt-out both look to be good additions.
However, given there's been solid progress on opentelemetry-util-genai since this was opened in October, I think it would be worth revisiting the community discussion above about aligning with that package before investing any more time / effort here.
A lot of of the content capture logic in this PR is what genai-utils is meant to surface.
A few other things worth addressing when this gets updated:
BaseCallbackHandlerProtocol/stub is duplicated across__init__.pyandcallback_handler.py# pyright: reportMissingImports=falseat the top of both files suppresses the type checker rather than fixing the importsOTEL_INSTRUMENTATION_LANGCHAIN_CAPTURE_MESSAGESdoesn't match the now-standardOTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENTenv var used by other GenAI instrumentations
MikeGoldsmith
left a comment
MikeGoldsmith
left a comment
There was a problem hiding this comment.
As previous review said, the genai-utils has done a lot of this work now so this will need updating to.
12 participants
Description
This PR renames the package to
opentelemetry-instrumentation-langchain-v2and updates the collector to the latest spec.This allows us to release the package as the official opentelemetry package.
Fixes # (issue)
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Does This PR Require a Core Repo Change?
Checklist:
See contributing.md for styleguide, changelog guidelines, and more.
Sample outputs
examples/manual/main.py:{ "name": "chat gpt-4.1", "context": { "trace_id": "1ec262108b837224724eab77bedceaae", "span_id": "bc78d3a82d35ca59" }, "parent_id": null, "start_time": "2025-10-20T20:06:17.074000Z", "end_time": "2025-10-20T20:06:18.047000Z", "status": { "code": 0, "message": null }, "kind": "CLIENT", "events": [], "attributes": { "gen_ai.operation.name": "chat", "gen_ai.request.model": "gpt-4.1", "gen_ai.provider.name": "openai", "gen_ai.request.top_p": "0.9", "gen_ai.request.frequency_penalty": "0.5", "gen_ai.request.presence_penalty": "0.5", "gen_ai.request.stop_sequences": "{\"values\":[{\"stringValue\":\"\\n\",\"value\":\"string_value\"},{\"stringValue\":\"Human:\",\"value\":\"string_value\"},{\"stringValue\":\"AI:\",\"value\":\"string_value\"}]}", "gen_ai.request.seed": "100", "gen_ai.request.temperature": "0.1", "gen_ai.request.max_tokens": "100", "gen_ai.input.messages": "[{\"type\": \"system\", \"content\": \"You are a helpful assistant!\", \"additional_kwargs\": {}, \"response_metadata\": {}}, {\"type\": \"human\", \"content\": \"What is the capital of France?\", \"additional_kwargs\": {}, \"response_metadata\": {}}]", "gen_ai.usage.input_tokens": "24", "gen_ai.usage.output_tokens": "7", "gen_ai.response.finish_reasons": "{\"values\":[{\"stringValue\":\"stop\",\"value\":\"string_value\"}]}", "gen_ai.response.model": "gpt-4.1-2025-04-14", "gen_ai.response.id": "chatcmpl-CSqQzLwwhz6w65N2px0zEtaOESlY8", "openai.response.service_tier": "default", "openai.response.system_fingerprint": "fp_e24a1fec47", "gen_ai.output.messages": "[{\"type\": \"ai\", \"content\": \"The capital of France is Paris.\", \"additional_kwargs\": {\"refusal\": null}, \"response_metadata\": {\"token_usage\": {\"completion_tokens\": 7, \"prompt_tokens\": 24, \"total_tokens\": 31, \"completion_tokens_details\": {\"accepted_prediction_tokens\": 0, \"audio_tokens\": 0, \"reasoning_tokens\": 0, \"rejected_prediction_tokens\": 0}, \"prompt_tokens_details\": {\"audio_tokens\": 0, \"cached_tokens\": 0}}, \"model_name\": \"gpt-4.1-2025-04-14\", \"system_fingerprint\": \"fp_e24a1fec47\", \"id\": \"chatcmpl-CSqQzLwwhz6w65N2px0zEtaOESlY8\", \"service_tier\": \"default\", \"finish_reason\": \"stop\", \"logprobs\": null}, \"tool_calls\": [], \"usage_metadata\": {\"input_tokens\": 24, \"output_tokens\": 7, \"total_tokens\": 31, \"input_token_details\": {\"audio\": 0, \"cache_read\": 0}, \"output_token_details\": {\"audio\": 0, \"reasoning\": 0}}, \"id\": \"run--f1ee6e87-819b-4f9e-b630-708b50c1d101-0\"}]" }, "resource": { "attributes": { "telemetry.sdk.language": "python", "telemetry.sdk.name": "opentelemetry", "telemetry.sdk.version": "1.38.0", "service.name": "opentelemetry-python-langchain-manual" }, "schema_url": null }, "links": [] }examples/multi_agent_travel_planner/main.py: to show invoke agent, execute tool :root span:
{ "name": "invoke_agent travel_multi_agent_planner", "context": { "trace_id": "0b1b755234fb5bd27ddddfc7e357c287", "span_id": "3cd74ba3e76a7432" }, "parent_id": null, "start_time": "2025-10-20T20:06:50.256000Z", "end_time": "2025-10-20T20:07:14.553000Z", "status": { "code": 0, "message": null }, "kind": "CLIENT", "events": [], "attributes": { "gen_ai.operation.name": "invoke_agent", "gen_ai.provider.name": "openai", "gen_ai.request.model": "gpt-4.1", "gen_ai.agent.name": "travel_multi_agent_planner", "gen_ai.agent.id": "travel_planner_f07876ef-2f3e-485a-9da6-4fff4b6b234b", "gen_ai.conversation.id": "f07876ef-2f3e-485a-9da6-4fff4b6b234b", "gen_ai.request.temperature": "0.4", "gen_ai.request.top_p": "1", "gen_ai.request.max_tokens": "1024", "gen_ai.request.frequency_penalty": "0", "gen_ai.request.presence_penalty": "0", "server.address": "api.openai.com", "server.port": "443", "service.name": "opentelemetry-python-langchain-multi-agent", "gen_ai.input.messages": "[{\"role\": \"user\", \"parts\": [{\"type\": \"text\", \"content\": \"We're planning a romantic long-week trip to Paris from Seattle next month. We'd love a boutique hotel, business-class flights and a few unique experiences.\"}]}]", "gen_ai.response.model": "gpt-4.1", "gen_ai.output.messages": "[{\"role\": \"assistant\", \"parts\": [{\"type\": \"text\", \"content\": \"**Romantic Paris Itinerary: Seattle to Paris** \\nDates: November 19\\u201326, 2025 | 2 Travelers\\n\\n---\\n\\n### Flights\\n- **Airline:** CloudNine (Non-stop)\\n- **Class:** Premium Economy (upgrade to business class available upon request)\\n- **Departure:** Nov 19, 09:15 from Seattle\\n- **Arrival:** Nov 19, 17:05 in Paris\\n- **Return:** Nov 26, direct flight to Seattle\\n- **Fare:** $717 per person (return)\\n\\n---\\n\\n### Accommodation\\n- **Hotel:** The Atlas (Boutique hotel near historic centre)\\n- **Room:** Stylish suite with breakfast included\\n- **Features:** Rooftop bar, intimate atmosphere\\n- **Rate:** $370/night\\n\\n---\\n\\n### Unique Experiences\\n1. **Eiffel Tower at Sunset** \\n Witness Paris transform as the city lights up\\u2014perfect for a romantic evening.\\n\\n2. **Seine Dinner Cruise** \\n Savor gourmet French cuisine while cruising past illuminated landmarks.\\n\\n3. **Day Trip to Versailles** \\n Explore the magnificent palace and gardens\\u2014an unforgettable royal escape.\\n\\n---\\n\\n**Summary:** \\nEnjoy a seamless, romantic week in Paris with non-stop premium flights, a chic boutique hotel, and curated experiences that capture the city\\u2019s magic.\"}], \"finish_reason\": \"stop\"}]", "metadata.final_plan.preview": "**Romantic Paris Itinerary: Seattle to Paris** \nDates: November 19–26, 2025 | 2 Travelers\n\n---\n\n### Flights\n- **Airline:** CloudNine (Non-stop)\n- **Class:** Premium Economy (upgrade to business class available upon request)\n- **Departure:** Nov 19, 09:15 from Seattle\n- **Arrival:** Nov 19, 17:05 in Paris\n- **Return:** Nov 26, direct flight to Seattle\n- **Fare:** $717 per person (return)\n\n---\n\n### Accommodation\n- **Hotel:** The Atlas (Boutique hotel near historic centre)\n- **Room:** Stylish suit...", "metadata.session_id": "f07876ef-2f3e-485a-9da6-4fff4b6b234b", "metadata.agents_used": "3" }, "resource": { "attributes": { "telemetry.sdk.language": "python", "telemetry.sdk.name": "opentelemetry", "telemetry.sdk.version": "1.38.0", "service.name": "opentelemetry-python-langchain-multi-agent" }, "schema_url": null }, "links": [] }chat span:
{ "name": "chat gpt-4.1", "context": { "trace_id": "0b1b755234fb5bd27ddddfc7e357c287", "span_id": "ea4d36857fec03d4" }, "parent_id": "3cd74ba3e76a7432", "start_time": "2025-10-20T20:06:59.323000Z", "end_time": "2025-10-20T20:07:00.636000Z", "status": { "code": 0, "message": null }, "kind": "CLIENT", "events": [], "attributes": { "gen_ai.operation.name": "chat", "gen_ai.request.model": "gpt-4.1", "gen_ai.provider.name": "openai", "gen_ai.request.temperature": "0.4", "gen_ai.tool.definitions": "[{\"type\": \"function\", \"function\": {\"name\": \"mock_search_flights\", \"description\": \"Return mock flight options for a given origin/destination pair.\", \"parameters\": {\"properties\": {\"origin\": {\"type\": \"string\"}, \"destination\": {\"type\": \"string\"}, \"departure\": {\"type\": \"string\"}}, \"required\": [\"origin\", \"destination\", \"departure\"], \"type\": \"object\"}}}]", "gen_ai.input.messages": "[{\"type\": \"human\", \"content\": \"Find an appealing flight from Seattle to Paris departing 2025-11-19 for 2 travellers.\", \"additional_kwargs\": {}, \"response_metadata\": {}, \"id\": \"38d63232-442c-4e80-926d-61aef1a5db04\"}]", "gen_ai.usage.input_tokens": "80", "gen_ai.usage.output_tokens": "29", "gen_ai.response.finish_reasons": "{\"values\":[{\"stringValue\":\"tool_calls\",\"value\":\"string_value\"}]}", "gen_ai.response.model": "gpt-4.1-2025-04-14", "gen_ai.response.id": "chatcmpl-CSqRf8S10ZfTPRXllsCm5224XOE8k", "openai.response.service_tier": "default", "openai.response.system_fingerprint": "fp_e24a1fec47", "gen_ai.output.messages": "[{\"type\": \"ai\", \"content\": \"\", \"additional_kwargs\": {\"tool_calls\": [{\"id\": \"call_BbDd8whyH8RxFOVOF5LxMkLL\", \"function\": {\"arguments\": \"{\\\"origin\\\":\\\"Seattle\\\",\\\"destination\\\":\\\"Paris\\\",\\\"departure\\\":\\\"2025-11-19\\\"}\", \"name\": \"mock_search_flights\"}, \"type\": \"function\"}], \"refusal\": null}, \"response_metadata\": {\"token_usage\": {\"completion_tokens\": 29, \"prompt_tokens\": 80, \"total_tokens\": 109, \"completion_tokens_details\": {\"accepted_prediction_tokens\": 0, \"audio_tokens\": 0, \"reasoning_tokens\": 0, \"rejected_prediction_tokens\": 0}, \"prompt_tokens_details\": {\"audio_tokens\": 0, \"cached_tokens\": 0}}, \"model_name\": \"gpt-4.1-2025-04-14\", \"system_fingerprint\": \"fp_e24a1fec47\", \"id\": \"chatcmpl-CSqRf8S10ZfTPRXllsCm5224XOE8k\", \"service_tier\": \"default\", \"finish_reason\": \"tool_calls\", \"logprobs\": null}, \"tool_calls\": [{\"name\": \"mock_search_flights\", \"args\": {\"origin\": \"Seattle\", \"destination\": \"Paris\", \"departure\": \"2025-11-19\"}, \"id\": \"call_BbDd8whyH8RxFOVOF5LxMkLL\", \"type\": \"tool_call\"}], \"usage_metadata\": {\"input_tokens\": 80, \"output_tokens\": 29, \"total_tokens\": 109, \"input_token_details\": {\"audio\": 0, \"cache_read\": 0}, \"output_token_details\": {\"audio\": 0, \"reasoning\": 0}}, \"id\": \"run--39786f05-a793-4c60-bbce-ebc40c00a935-0\"}]" }, "resource": { "attributes": { "telemetry.sdk.language": "python", "telemetry.sdk.name": "opentelemetry", "telemetry.sdk.version": "1.38.0", "service.name": "opentelemetry-python-langchain-multi-agent" }, "schema_url": null }, "links": [] }execute tool span:
{ "name": "execute_tool mock_search_flights", "context": { "trace_id": "0b1b755234fb5bd27ddddfc7e357c287", "span_id": "34cf2242947a1664" }, "parent_id": "3cd74ba3e76a7432", "start_time": "2025-10-20T20:07:00.639000Z", "end_time": "2025-10-20T20:07:00.640000Z", "status": { "code": 0, "message": null }, "kind": "INTERNAL", "events": [], "attributes": { "gen_ai.operation.name": "execute_tool", "gen_ai.tool.name": "mock_search_flights", "gen_ai.tool.call.arguments": "{\"origin\": \"Seattle\", \"destination\": \"Paris\", \"departure\": \"2025-11-19\"}", "gen_ai.tool.call.result": "{\"content\": \"Top choice: CloudNine non-stop service Seattle->Paris, depart 2025-11-19 09:15, arrive 2025-11-19 17:05. Premium economy fare $717 return.\", \"additional_kwargs\": {}, \"response_metadata\": {}, \"type\": \"tool\", \"name\": \"mock_search_flights\", \"id\": null, \"tool_call_id\": \"call_BbDd8whyH8RxFOVOF5LxMkLL\", \"artifact\": null, \"status\": \"success\"}" }, "resource": { "attributes": { "telemetry.sdk.language": "python", "telemetry.sdk.name": "opentelemetry", "telemetry.sdk.version": "1.38.0", "service.name": "opentelemetry-python-langchain-multi-agent" }, "schema_url": null }, "links": [] }