Skip to content

Commit 640c2a2

Browse files
authored
fix: add content capture via otel (#5060)
# What does this PR do? <!-- Provide a short summary of what this PR does and why. Link to relevant issues if applicable. --> <!-- If resolving an issue, uncomment and update the line below --> <!-- Closes #[issue-number] --> ## Test Plan <!-- Describe the tests you ran to verify your changes with result summaries. *Provide clear instructions so the plan can be easily re-executed.* --> <!-- For API changes, include: 1. A testing script (Python, curl, etc.) that exercises the new/modified endpoints 2. The output from running your script Example: ```python ... ... ``` Output: ``` <paste actual output here> ``` -->
1 parent 424ff0c commit 640c2a2

1 file changed

Lines changed: 71 additions & 0 deletions

File tree

scripts/telemetry/README.md

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -153,6 +153,77 @@ A pre-provisioned **Llama Stack** dashboard is available in Grafana with panels
153153
- P95 / P99 HTTP Server Duration
154154
- Total HTTP Requests
155155

156+
### GenAI message content capture (logs vs spans)
157+
158+
- `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` controls whether prompts/outputs/tool-call args are captured. **Default: false** (no content captured). Set to `true` to capture.
159+
- Captured content is emitted as **logs** (e.g., events like `gen_ai.user.message` and `gen_ai.choice`), with `trace_id`/`span_id` for correlation.
160+
- Spans carry structured metadata (model, finish_reason, usage tokens, latency, HTTP call, etc.) but **not** the raw text content.
161+
162+
### Exporter examples
163+
164+
**Local console (debugging):**
165+
```bash
166+
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true \
167+
OTEL_TRACES_EXPORTER=console \
168+
OTEL_LOGS_EXPORTER=console \
169+
opentelemetry-instrument python llama-stack-client.py
170+
```
171+
172+
**Send to Collector (recommended):**
173+
```bash
174+
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true \
175+
OTEL_TRACES_EXPORTER=otlp \
176+
OTEL_LOGS_EXPORTER=otlp \
177+
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
178+
opentelemetry-instrument python llama-stack-client.py
179+
```
180+
181+
### Jaeger caveat (logs)
182+
183+
- Jaeger’s built-in OTLP typically ingests **traces only**, not logs. If you point `OTEL_LOGS_EXPORTER=otlp` at Jaeger, logs will be rejected (e.g., 404) and you will not see message content.
184+
- To view captured content, send logs to an OTel Collector (or another backend that supports OTLP logs). For debugging, you can also use `OTEL_LOGS_EXPORTER=console` to print logs to stdout.
185+
186+
### Minimal Collector example (traces → Jaeger, logs → stdout)
187+
188+
```yaml
189+
receivers:
190+
otlp:
191+
protocols:
192+
http:
193+
grpc:
194+
195+
processors:
196+
batch: {}
197+
198+
exporters:
199+
jaeger:
200+
endpoint: localhost:14250
201+
tls:
202+
insecure: true
203+
logging: # view logs in collector stdout
204+
loglevel: info
205+
206+
service:
207+
pipelines:
208+
traces:
209+
receivers: [otlp]
210+
processors: [batch]
211+
exporters: [jaeger]
212+
logs:
213+
receivers: [otlp]
214+
processors: [batch]
215+
exporters: [logging]
216+
```
217+
218+
Use with:
219+
```bash
220+
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true \
221+
OTEL_TRACES_EXPORTER=otlp \
222+
OTEL_LOGS_EXPORTER=otlp \
223+
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
224+
opentelemetry-instrument python llama-stack-client.py
225+
```
226+
156227
## Cleanup
157228

158229
Stop and remove all telemetry containers:

0 commit comments

Comments
 (0)