Observability for agent systems is the discipline of capturing, visualizing, and interpreting every signal an LLM or agent produces: traces, spans, metrics, logs, and evaluation scores. Unlike traditional application monitoring, agent observability must reconstruct multi-step reasoning flows, tool calls, latency, cost, and correctness in a single coherent view.
Agents are not stateless request handlers. They plan, call tools, loop, fail, recover, and sometimes hallucinate. Without high-fidelity tracing and telemetry, debugging a failed multi-agent run is like reading a stack trace printed on confetti. A solid observability layer records each step, enriches it with metadata (cost, latency, model, prompt version), and feeds the Evaluation Layer with ground-truth data. It sits between the runtime and the evaluator: runtime produces spans, observability stores and visualizes them, and evaluation uses them to score quality, regressions, and drift. If evaluation tells you whether your agent is good, observability tells you why it is good—or why it broke.
Summary: Open observability and data visualization platform.
Why it matters: Grafana is the leading dashboard and visualization layer for metrics, logs, and traces. In an agent stack it is usually the front-end through which you view telemetry coming from Prometheus, Loki, Tempo, or OpenTelemetry backends.
Stats: ⭐ 75,647 · 🍴 14,361 · Last commit: 2026-07-15 · Activity: Active
Priority: 5/5 · Difficulty: Intermediate · Tags: alerting, analytics, business-intelligence, dashboard, data-visualization, elasticsearch, go, grafana
What to learn from it:
- How to build dashboards that correlate latency, error rate, and cost across agent traces.
- How to unify logs, metrics, and traces in a single pane without vendor lock-in.
- The role of data-source plugins and how they map to your own telemetry backend.
- Alerting patterns: when to alert on p99 latency versus failed tool-call ratios.
Summary: Open-source monitoring system and time series database.
Why it matters: Prometheus is the standard backend for metrics in observability stacks, including AI infrastructure monitoring. Agent systems emit counters, histograms, and gauges that Prometheus stores and queries.
Stats: ⭐ 65,231 · 🍴 10,753 · Last commit: 2026-07-15 · Activity: Active
Priority: 5/5 · Difficulty: Advanced · Tags: alerting, graphing, hacktoberfest, metrics, monitoring, prometheus, time-series
What to learn from it:
- Metric types and when to use counters vs. histograms for LLM request latency.
- Recording rules and aggregation for high-cardinality agent labels.
- Alertmanager patterns for noisy multi-step agent pipelines.
- Long-term storage trade-offs: Prometheus TSDB versus Thanos/Cortex/Mimir.
Summary: Open-source AI engineering platform for LLM observability, evals and prompt management.
Why it matters: Langfuse is the fastest-growing open observability layer for LLM and agent apps. It is OpenTelemetry-native and offers easy self-hosting, making it a must-learn for anyone shipping agent systems.
Stats: ⭐ 31,197 · 🍴 3,289 · Last commit: 2026-07-15 · Activity: Active
Priority: 5/5 · Difficulty: Intermediate · Tags: analytics, autogen, evaluation, langchain, large-language-models, llama-index, llm, llm-evaluation
What to learn from it:
- Trace-centric UX: how to represent a multi-step agent run as a nested trace.
- Prompt management and versioning integrated with observability.
- OpenTelemetry ingestion and how it unifies auto-instrumented and manual spans.
- Combining traces with evaluation scores for continuous regression detection.
Summary: Open-source AI/ML lifecycle and observability platform.
Why it matters: MLflow is the de-facto standard for ML experiments and model management, and it increasingly adds LLM and agent tracing, prompts, and evaluations.
Stats: ⭐ 27,040 · 🍴 6,001 · Last commit: 2026-07-15 · Activity: Active
Priority: 5/5 · Difficulty: Intermediate · Tags: agentops, agents, ai, ai-governance, evaluation, langchain, llm-evaluation, llmops
What to learn from it:
- Experiment tracking patterns that transfer from ML training to agent prompt iteration.
- Model registry workflows for promoting agent prompts or small fine-tuned models.
- Tracing integration: how MLflow captures LangChain/LLM calls as traces.
- Governance and lineage hooks that become mandatory once agents reach production.
Summary: Open-source observability, evaluation and monitoring for LLM, RAG and agent workflows.
Why it matters: Opik is Comet's fully open LLM observability stack with tracing, evals, and datasets. It has strong momentum and a simple local setup that makes it easy to adopt alongside Langfuse or Phoenix.
Stats: ⭐ 20,608 · 🍴 1,607 · Last commit: 2026-07-15 · Activity: Active
Priority: 5/5 · Difficulty: Intermediate · Tags: evaluation, hacktoberfest, hacktoberfest2025, langchain, llama-index, llm, llm-evaluation, llm-observability
What to learn from it:
- Dataset-driven evaluation workflows tied directly to production traces.
- How to compare traces across model versions and prompt variants.
- Lightweight self-hosting architecture and local-first development experience.
- Integration patterns with existing Comet experiment tracking if you already use it.
Summary: LLM evaluation framework.
Why it matters: DeepEval is a widely adopted Python framework for unit-test style LLM evals, integrated with CI and observability. It turns evaluation into a regular engineering test suite.
Stats: ⭐ 16,869 · 🍴 1,661 · Last commit: 2026-07-14 · Activity: Active
Priority: 4/5 · Difficulty: Intermediate · Tags: evaluation-framework, evaluation-metrics, llm-evaluation, llm-evaluation-framework, llm-evaluation-metrics, python
What to learn from it:
- Unit-test ergonomics for LLM metrics such as hallucination, answer relevance, and faithfulness.
- CI integration: how to gate merges on regression thresholds rather than gut feeling.
- Custom metric design and the trade-offs between LLM-as-a-judge and deterministic scorers.
- Why observability without evaluation is only half the story.
Summary: Evaluation framework for RAG and LLM applications.
Why it matters: Ragas is the reference implementation for RAG evaluation metrics used across the LLM observability landscape. If you ship RAG, you will eventually measure it with Ragas-style metrics.
Stats: ⭐ 14,850 · 🍴 1,564 · Last commit: 2026-02-24 · Activity: Moderate
Priority: 4/5 · Difficulty: Intermediate · Tags: evaluation, llm, llmops
What to learn from it:
- Metric definitions: context precision, context recall, faithfulness, and answer relevance.
- How to synthesize test sets from your own documents for continuous evaluation.
- The relationship between retrieval quality and generation quality in traces.
- When to use Ragas directly versus embedding its metrics in another observability tool.
Summary: Open-source AI observability and evaluation platform.
Why it matters: Phoenix has deep roots in ML monitoring plus modern LLM tracing and evals. It is widely used for RAG and agent debugging, especially where embeddings and retrieval quality matter.
Stats: ⭐ 10,571 · 🍴 986 · Last commit: 2026-07-15 · Activity: Active
Priority: 4/5 · Difficulty: Intermediate · Tags: agents, ai-monitoring, ai-observability, aiengineering, anthropic, datasets, evals, langchain
What to learn from it:
- Embedding visualization and drift detection for RAG retrievers.
- Trace-based debugging of agent loops with span-level annotations.
- A/B evaluation of prompt and model variants against production traces.
- How a tool born in ML monitoring evolves to serve LLM workloads.
Summary: OpenTelemetry-based observability instrumentation for GenAI/LLM apps.
Why it matters: OpenLLMetry provides automatic instrumentation for popular LLM SDKs and frameworks using OpenTelemetry. It is the bridge between your application code and any OTel-compatible observability backend.
Stats: ⭐ 7,302 · 🍴 1,018 · Last commit: 2026-07-13 · Activity: Active
Priority: 4/5 · Difficulty: Intermediate · Tags: artifical-intelligence, datascience, generative-ai, good-first-issue, good-first-issues, help-wanted, llm, llmops
What to learn from it:
- How auto-instrumentation captures spans without code changes in supported frameworks.
- OTel semantic conventions for GenAI and why consistency matters.
- The split between instrumentation and backend: capture once, send anywhere.
- Best practices for sampling high-volume agent traces without losing critical failures.
Summary: Official OpenTelemetry Go API and SDK.
Why it matters: This is critical for Go-based AI services and agents that need portable, vendor-neutral telemetry. It provides the building blocks for traces, metrics, and logs in a growing language ecosystem.
Stats: ⭐ 6,465 · 🍴 1,417 · Last commit: 2026-07-15 · Activity: Active
Priority: 4/5 · Difficulty: Intermediate · Tags: logging, metrics, opentelemetry, tracing
What to learn from it:
- How to instrument a Go agent runtime with traces and custom span attributes.
- Context propagation across goroutines and external HTTP calls.
- Exporter configuration: OTLP, console, and vendor backends.
- Why vendor-neutral instrumentation lets you swap observability backends without rewriting code.
No repositories are currently classified as Emerging in this category. This section will be updated as the landscape evolves.
No repositories are currently classified as Historical in this category.
| Repo | Stars | Language | License | Last Commit | Activity | Difficulty | Ranking |
|---|---|---|---|---|---|---|---|
| grafana/grafana | 75,647 | TypeScript | AGPL-3.0 | 2026-07-15 | Active | Intermediate | must_learn |
| prometheus/prometheus | 65,231 | Go | Apache-2.0 | 2026-07-15 | Active | Advanced | must_learn |
| langfuse/langfuse | 31,197 | TypeScript | NOASSERTION | 2026-07-15 | Active | Intermediate | must_learn |
| mlflow/mlflow | 27,040 | Python | Apache-2.0 | 2026-07-15 | Active | Intermediate | must_learn |
| comet-ml/opik | 20,608 | Python | Apache-2.0 | 2026-07-15 | Active | Intermediate | must_learn |
| confident-ai/deepeval | 16,869 | Python | Apache-2.0 | 2026-07-14 | Active | Intermediate | strongly_recommended |
| explodinggradients/ragas | 14,850 | Python | Apache-2.0 | 2026-02-24 | Moderate | Intermediate | strongly_recommended |
| arize-ai/phoenix | 10,571 | Python | NOASSERTION | 2026-07-15 | Active | Intermediate | strongly_recommended |
| traceloop/openllmetry | 7,302 | Python | Apache-2.0 | 2026-07-13 | Active | Intermediate | strongly_recommended |
| open-telemetry/opentelemetry-go | 6,465 | Go | Apache-2.0 | 2026-07-15 | Active | Intermediate | strongly_recommended |
flowchart LR
subgraph Runtime
A[Agent / LLM App]
end
subgraph Instrumentation
I[SDK / Auto-instrumentation<br/>OpenLLMetry / OTel]
end
subgraph Backend
OT[OpenTelemetry Collector]
LF[Langfuse / Phoenix / Opik]
P[(Prometheus)]
G[Grafana]
end
subgraph Evaluation
EV[Evaluation Layer]
end
A -->|spans, metrics, logs| I
I --> OT
OT --> LF
OT --> P
LF -->|scores, traces| EV
P -->|metrics| G
LF -->|dashboards| G
EV -->|feedback| A
- langfuse/langfuse — start with a purpose-built LLM observability platform to understand trace-centric debugging.
- traceloop/openllmetry — learn how OpenTelemetry instrumentation captures spans automatically.
- comet-ml/opik — compare an alternative full-stack observability tool and its dataset-driven evaluation.
- arize-ai/phoenix — add embedding and RAG-specific debugging to your toolkit.
- prometheus/prometheus — master the metrics backend that powers production alerting.
- grafana/grafana — visualize everything and build operational dashboards.
- mlflow/mlflow — connect experiment tracking, model lineage, and governance.
- confident-ai/deepeval — turn traces into testable evaluation metrics.
- explodinggradients/ragas — specialize in RAG evaluation metrics.
- open-telemetry/opentelemetry-go — instrument a Go-based agent runtime from scratch.
- Langfuse when you want a modern, OpenTelemetry-native, self-hostable LLM observability hub with prompt management.
- Opik when you value Comet's experiment ecosystem and a lightweight local setup.
- Phoenix when your problem is RAG/embedding-centric and you need embedding drift and retrieval debugging.
- OpenLLMetry when you already have an OTel pipeline and want automatic instrumentation for LLM frameworks.
- Prometheus + Grafana when you need classic infrastructure metrics, SLIs, and operational dashboards for agent services.
- MLflow when governance, model lineage, and cross-team ML lifecycle management matter as much as live traces.
- DeepEval / Ragas when you want to convert observability data into automated, CI-gated evaluation tests.
- OpenTelemetry Go when your agent runtime is written in Go and you need vendor-neutral telemetry.
- ← Back to README
- → Evaluation Layer — observability feeds scores and regressions into the evaluation layer