evals.md

Evaluation Layer

Evaluation is the highest-leverage activity in agent engineering: a small, repeatable test beats a thousand hunches about what your prompt or agent will do. This page treats prompt evaluation, agent evaluation, and Evaluation-Driven Development (EDD) as a single discipline because they share the same loop: define expectations, run the system, capture traces, score outputs, and iterate.

Why this is the centerpiece

Without evals, every prompt tweak is a blind bet and every agent release is a hope. With evals, guesswork becomes measurable progress: regression tests catch prompt drift before users do, A/B tests compare variants with numbers instead of vibes, and multi-turn trajectory scoring reveals how agents actually behave across a conversation. Evaluation-Driven Development closes the loop by wiring these tests into CI/CD, so every code or prompt change is validated like any other software artifact. The result is not just better agents, but a team that can ship confidently and debug scientifically.

flowchart LR
    A[Dataset / Eval Case] --> B[Prompt or Agent Run]
    B --> C[Trace Capture]
    C --> D[Assertions + LLM Judge]
    D --> E[Aggregated Score]
    E --> F{Gates}
    F --> G[Regression]
    F --> H[A/B Test]
    F --> I[CI for AI]
    G --> A
    H --> A
    I --> A

Loading

Prompt Evaluation

Prompt evaluation is the foundation. If your prompts regress, everything built on top of them regresses too. These tools help you version prompts, assert against outputs, and gate changes in CI.

Must Learn

promptfoo/promptfoo

Declarative prompt, agent, and RAG testing and red-teaming CLI.

• Why it matters: The most adopted open-source prompt evaluation tool; used by OpenAI and Anthropic for regression tests, A/B comparisons, and CI/CD gates.
• Stats: ⭐ 23,284 · 🍴 2,088 · last commit 2026-07-15 · Active
• Learning priority / difficulty: 5 / Beginner
• Tags: cli ci-cd prompt-testing red-teaming llm-eval
• What to learn from it:
  • How to write YAML/JSON test configs that compare prompts variants side by side.
  • Using built-in assertions (contains, JSON schema, similarity, LLM-as-judge) to catch regressions.
  • Running red-team scans for jailbreaks and prompt injection before shipping.
  • Integrating promptfoo eval into GitHub Actions for prompt CI gates.
  • The difference between deterministic assertions and model-graded rubrics.

langfuse/langfuse

Open-source AI engineering platform: evals, observability, prompt mgmt.

• Why it matters: Widely deployed open-source LLMOps suite that ties prompt versioning, datasets, and evaluation metrics together in one self-hosted platform.
• Stats: ⭐ 31,197 · 🍴 3,289 · last commit 2026-07-15 · Active
• Learning priority / difficulty: 5 / Intermediate
• Tags: llmops prompt-management observability evaluation
• What to learn from it:
  • How prompt versioning and dataset management fit into the same workflow.
  • Capturing traces and attaching scores to specific prompt versions.
  • Self-hosting an LLMOps platform and keeping data out of SaaS pipelines.
  • Building dashboards that show prompt performance over time.
  • Connecting production traces back to evaluation datasets for closed-loop improvement.

openai/evals

Framework and open registry for evaluating LLMs and LLM systems.

• Why it matters: The canonical benchmark framework from OpenAI; historically important for standardizing LLM evals and still the go-to registry for public benchmarks.
• Stats: ⭐ 18,922 · 🍴 3,025 · last commit 2026-04-14 · Moderate
• Learning priority / difficulty: 5 / Intermediate
• Tags: benchmarks llm-eval openai
• What to learn from it:
  • The registry pattern for sharing evals across teams and models.
  • How to define a new eval with prompts, expected behavior, and metrics.
  • The trade-offs between public benchmarks and task-specific prompt tests.
  • Why benchmark saturation does not mean your prompt works in production.

Strongly Recommended

mlflow/mlflow

Open-source AI engineering platform for agents, LLMs, and ML models.

• Why it matters: While broader than prompts, MLflow's LLM tracking, prompt registry, and evaluation APIs make it a production-grade choice for prompt A/B testing.
• Stats: ⭐ 27,040 · 🍴 6,001 · last commit 2026-07-15 · Active
• Learning priority / difficulty: 4 / Intermediate
• Tags: mlops llmops tracking prompt-registry
• What to learn from it:
  • How experiment tracking extends from ML models to LLM prompts.
  • Using the prompt registry to version and deploy prompt variants.
  • Comparing evaluation runs with MLflow's UI and metric logging.
  • Integrating LLM evaluation APIs into an existing MLflow workflow.

comet-ml/opik

Debug, evaluate, and monitor LLM apps with tracing and dashboards.

• Why it matters: Fast-growing Comet evaluation layer with strong prompt-version tracking, automated scoring, and integrations with popular LLM frameworks.
• Stats: ⭐ 20,608 · 🍴 1,607 · last commit 2026-07-15 · Active
• Learning priority / difficulty: 4 / Intermediate
• Tags: tracing evaluation prompt-engineering observability
• What to learn from it:
  • Setting up automated evals that score prompts on every run.
  • How tracing data feeds back into prompt improvement loops.
  • Using dashboards to compare prompt versions and model behavior.
  • Integrating with LangChain and other frameworks for end-to-end visibility.

confident-ai/deepeval

The LLM Evaluation Framework.

• Why it matters: Python-first evaluation framework with built-in metrics, synthetic data generation, and pytest integration for prompt regression testing.
• Stats: ⭐ 16,869 · 🍴 1,661 · last commit 2026-07-14 · Active
• Learning priority / difficulty: 4 / Intermediate
• Tags: python llm-eval pytest metrics
• What to learn from it:
  • Writing pytest-style evals that feel like unit tests.
  • Applying metrics such as answer relevancy, faithfulness, and contextual recall.
  • Generating synthetic test data from production examples.
  • Running evals locally and in CI with minimal setup.

arize-ai/phoenix

AI observability and evaluation.

• Why it matters: Open-source observability/eval platform from Arize with strong prompt/response tracing, embeddings analysis, and eval-driven workflows.
• Stats: ⭐ 10,571 · 🍴 986 · last commit 2026-07-15 · Active
• Learning priority / difficulty: 4 / Advanced
• Tags: observability llm-eval embeddings tracing
• What to learn from it:
  • How embeddings and retrieval analysis inform prompt quality.
  • Running evals directly on traced prompt/response pairs.
  • Using Phoenix for RAG-specific prompt diagnostics.
  • Building eval-driven notebooks for exploratory analysis.

agenta-ai/agenta

Open-source LLMOps platform: prompt playground, management, evaluation.

• Why it matters: Provides a prompt playground, versioning, and evaluation workflows specifically built for comparing prompt variants and LLM outputs.
• Stats: ⭐ 4,295 · 🍴 568 · last commit 2026-07-15 · Active
• Learning priority / difficulty: 4 / Intermediate
• Tags: prompt-playground llmops evaluation prompt-management
• What to learn from it:
  • How a prompt playground accelerates variant comparison.
  • Versioning prompts and linking them to evaluation results.
  • Deploying the winning prompt variant to production.
  • Collaboration workflows for prompt engineers and developers.

truera/trulens

Evaluation and tracking for LLM experiments and AI agents.

• Why it matters: Provides feedback functions and tracking for prompt-level evaluation, particularly strong for RAG and agentic applications.
• Stats: ⭐ 3,438 · 🍴 310 · last commit 2026-07-11 · Active
• Learning priority / difficulty: 3 / Intermediate
• Tags: llm-eval rag agents feedback
• What to learn from it:
  • Using feedback functions as composable evaluators.
  • Tracking which prompt versions produce the best feedback scores.
  • Combining RAG retrieval metrics with response quality metrics.
  • Instrumenting agents and apps for end-to-end evaluation.

Emerging

hegelai/prompttools

Open-source tools for prompt testing and experimentation.

• Why it matters: Lightweight Python toolkit for side-by-side prompt/model comparison and vector-store testing; useful for quick prompt experiments.
• Stats: ⭐ 3,041 · 🍴 256 · last commit 2026-02-11 · Moderate
• Learning priority / difficulty: 3 / Beginner
• Tags: prompt-testing python experimentation vector-db
• What to learn from it:
  • Running quick prompt experiments in notebooks.
  • Comparing model outputs across providers without heavy infrastructure.
  • Testing vector store retrieval alongside prompt output.
  • When a lightweight tool is enough and when to graduate to promptfoo or Langfuse.

Agent Evaluation

Agent evaluation raises the stakes from single-turn outputs to multi-step behavior. You need to score not just what an agent says, but what it does across a trajectory.

Must Learn

promptfoo/promptfoo

Open-source LLM prompt, agent, and red-teaming evaluation platform.

• Why it matters: Used by OpenAI and Anthropic, it provides declarative YAML/JSON configs for testing prompts, agents, and RAG systems with CI/CD integration and security scanning.
• Stats: ⭐ 23,285 · 🍴 2,088 · last commit 2026-07-15 · Active
• Learning priority / difficulty: 5 / Intermediate
• Tags: llm-evaluation agent-evaluation red-teaming prompt-testing ci-cd
• What to learn from it:
  • Configuring agent evals with tools, state, and multi-turn expectations.
  • Running adversarial tests against tool-using agents.
  • Comparing agent variants with the same scenario set.
  • Using promptfoo as a CI gate for agent behavior.
  • Balancing deterministic checks with model-graded rubrics for agent outputs.

comet-ml/opik

Debug, evaluate, and monitor LLM applications, RAG systems, and agentic workflows.

• Why it matters: Opik combines comprehensive tracing, automated evaluations, and production dashboards, making it a strong open-source alternative to closed evaluation/observability stacks.
• Stats: ⭐ 20,608 · 🍴 1,607 · last commit 2026-07-15 · Active
• Learning priority / difficulty: 5 / Intermediate
• Tags: llm-evaluation tracing observability agent-evaluation rag
• What to learn from it:
  • Capturing full agent traces including tool calls and intermediate steps.
  • Scoring agent trajectories with built-in and custom metrics.
  • Setting up production dashboards for agent monitoring.
  • Correlating agent failures with specific trace segments.

openai/evals

OpenAI's framework and registry for evaluating LLMs and LLM systems.

• Why it matters: Created by OpenAI, it standardizes benchmark definitions and provides a large registry of community evals, serving as a reference for evaluating model and agent capabilities.
• Stats: ⭐ 18,922 · 🍴 3,025 · last commit 2026-04-14 · Moderate
• Learning priority / difficulty: 5 / Intermediate
• Tags: llm-evaluation benchmarks openai registry
• What to learn from it:
  • How community benchmarks map to agent capability evaluation.
  • Defining agent tasks as evals with clear success criteria.
  • The limits of single-shot benchmarks for multi-turn agents.
  • How to contribute a new agent eval to the registry.

confident-ai/deepeval

Open-source LLM evaluation framework with built-in metrics and test suites.

• Why it matters: DeepEval packages LLM-as-a-judge metrics (G-Eval, faithfulness, answer relevancy, etc.) into pytest-like workflows, making it easy to evaluate agents and RAG pipelines.
• Stats: ⭐ 16,869 · 🍴 1,661 · last commit 2026-07-14 · Active
• Learning priority / difficulty: 5 / Intermediate
• Tags: llm-evaluation llm-as-judge rag-evaluation metrics
• What to learn from it:
  • Applying G-Eval and other model-graded metrics to agent outputs.
  • Building test suites that cover normal, edge, and failure cases.
  • Using pytest patterns to evaluate agents like any other Python module.
  • Combining RAG metrics with agent task-completion metrics.

Strongly Recommended

explodinggradients/ragas

Framework for evaluating RAG and LLM applications with reference-free metrics.

• Why it matters: Ragas popularized reference-free evaluation of RAG pipelines and offers metrics like faithfulness, answer relevancy, and context precision that are widely adopted.
• Stats: ⭐ 14,850 · 🍴 1,564 · last commit 2026-02-24 · Moderate
• Learning priority / difficulty: 4 / Intermediate
• Tags: rag-evaluation llm-evaluation rag metrics
• What to learn from it:
  • Evaluating RAG agents without ground-truth answers.
  • Using faithfulness and context precision to debug retrieval.
  • When reference-free metrics are sufficient and when you need labeled data.
  • Integrating Ragas into an agent evaluation pipeline.

EleutherAI/lm-evaluation-harness

Framework for few-shot evaluation of language models on hundreds of benchmarks.

• Why it matters: The harness is a standard tool in academic and industry research for comparing base LLM performance across a wide range of tasks.
• Stats: ⭐ 13,289 · 🍴 3,409 · last commit 2026-07-13 · Active
• Learning priority / difficulty: 4 / Advanced
• Tags: llm-evaluation benchmarks few-shot academic
• What to learn from it:
  • Running hundreds of benchmarks against a base model.
  • Understanding few-shot prompting and task formatting.
  • Comparing pre-trained models before adding agent logic.
  • Extending the harness with custom tasks.

arize-ai/phoenix

AI observability and evaluation platform for LLM, agent, and computer-vision applications.

• Why it matters: Phoenix provides open-source tracing, evaluation, and experimentation tools specifically designed for agentic and RAG applications.
• Stats: ⭐ 10,571 · 🍴 986 · last commit 2026-07-15 · Active
• Learning priority / difficulty: 4 / Intermediate
• Tags: ai-observability llm-evaluation tracing agent-evaluation rag
• What to learn from it:
  • Instrumenting agents to capture tool calls, retrievals, and final outputs.
  • Evaluating agent trajectories with span-level metrics.
  • Using Phoenix experiments to compare agent configurations.
  • Diagnosing retrieval and reasoning failures in agent traces.

SWE-bench/SWE-bench

Benchmark for evaluating LLMs on real-world GitHub issue resolution.

• Why it matters: SWE-bench is the de facto benchmark for coding agents, measuring whether models can autonomously resolve real open-source software issues.
• Stats: ⭐ 5,427 · 🍴 922 · last commit 2026-04-01 · Moderate
• Learning priority / difficulty: 4 / Advanced
• Tags: coding-agents benchmark software-engineering llm-evaluation
• What to learn from it:
  • How to run a coding agent benchmark against real GitHub issues.
  • The gap between benchmark scores and production software engineering.
  • Setting up containerized evaluation environments for agents.
  • Measuring pass rates, patch correctness, and issue resolution.

UKGovernmentBEIS/inspect_ai

Framework for large language model evaluations from the UK AI Safety Institute.

• Why it matters: Inspect AI is built for robust, reproducible evaluations of agentic systems, with strong support for multi-turn tasks, scorers, and human-in-the-loop review.
• Stats: ⭐ 2,352 · 🍴 603 · last commit 2026-07-15 · Active
• Learning priority / difficulty: 4 / Advanced
• Tags: llm-evaluation agent-evaluation ai-safety multi-turn benchmarks
• What to learn from it:
  • Designing multi-turn evals with custom scorers.
  • Running evaluations with human-in-the-loop review.
  • Building reproducible evals for safety-critical agents.
  • Using solvers and scorers to separate agent logic from evaluation logic.

langchain-ai/langsmith-sdk

LangSmith client SDK for tracing, evaluating, and monitoring LangChain applications.

• Why it matters: LangSmith is widely used with LangChain/LangGraph agents; this SDK enables tracing, dataset management, and eval runs for agent workflows.
• Stats: ⭐ 968 · 🍴 260 · last commit 2026-07-15 · Active
• Learning priority / difficulty: 4 / Intermediate
• Tags: langchain tracing llm-evaluation agent-evaluation sdk
• What to learn from it:
  • Tracing LangChain/LangGraph agent runs end to end.
  • Creating datasets from production traces for evals.
  • Running eval runs against LangChain agent variants.
  • Integrating evaluation into the LangSmith hosted or self-hosted UI.

Evaluation-Driven Development

EDD closes the loop: your evals live in version control, run in CI, and guide every code and prompt change. It is the practice of treating evaluation as a first-class engineering workflow rather than an afterthought.

Must Learn

langfuse/langfuse

Open-source AI engineering platform for LLM evals, observability, metrics, prompt management, and datasets.

• Why it matters: Langfuse is one of the most widely adopted open-source LLM observability and evaluation platforms, providing an integrated stack for tracing, scoring, and improving AI applications in production.
• Stats: ⭐ 31,200 · 🍴 3,289 · last commit 2026-07-15 · Active
• Learning priority / difficulty: 5 / Intermediate
• Tags: llm-observability evaluation prompt-management tracing self-hosted
• What to learn from it:
  • Hosting an end-to-end evaluation and observability stack.
  • Connecting traces, scores, datasets, and prompt versions in one place.
  • Using Langfuse as the backbone of an EDD workflow.
  • Building feedback loops from production monitoring back to eval dataset creation.

promptfoo/promptfoo

Test prompts, agents, and RAGs with red teaming, pentesting, and CI/CD integration.

• Why it matters: Promptfoo has become the de facto standard for declarative prompt and agent evaluation, used by OpenAI and Anthropic for regression testing and red teaming.
• Stats: ⭐ 23,285 · 🍴 2,088 · last commit 2026-07-15 · Active
• Learning priority / difficulty: 5 / Intermediate
• Tags: prompt-testing red-teaming rag-evaluation ci-cd llm-eval
• What to learn from it:
  • Declaring evals in files that can be version-controlled and reviewed.
  • Running evals in CI with pass/fail gates.
  • Red-teaming and pentesting as part of the development cycle.
  • Treating prompt and agent changes like code changes.

confident-ai/deepeval

The LLM Evaluation Framework with built-in metrics and CI/CD integration.

• Why it matters: DeepEval provides a batteries-included Python framework for unit testing and evaluating LLM applications, making it easy to adopt eval-driven development workflows.
• Stats: ⭐ 16,868 · 🍴 1,661 · last commit 2026-07-14 · Active
• Learning priority / difficulty: 5 / Intermediate
• Tags: llm-evaluation evaluation-metrics python ci-cd rag
• What to learn from it:
  • Writing pytest evals that fail CI when quality drops.
  • Using built-in metrics as a starting point for custom ones.
  • Generating synthetic test cases from production data.
  • Integrating DeepEval with existing Python CI pipelines.

EleutherAI/lm-evaluation-harness

A framework for few-shot evaluation of language models.

• Why it matters: The EleutherAI harness is the canonical open-source tool for benchmarking foundation models across hundreds of tasks, serving as a reference implementation for model evaluation.
• Stats: ⭐ 13,289 · 🍴 3,409 · last commit 2026-07-13 · Active
• Learning priority / difficulty: 5 / Advanced
• Tags: model-evaluation benchmarks few-shot foundation-models nlp
• What to learn from it:
  • Standardizing model evaluation before building agents on top.
  • Running reproducible benchmarks across model versions.
  • Understanding task formats and few-shot templates.
  • Using benchmark results to drive model-selection decisions.

Strongly Recommended

mlflow/mlflow

Open-source AI engineering platform for debugging, evaluating, monitoring, and optimizing AI applications.

• Why it matters: MLflow provides mature experiment tracking, model registry, and evaluation capabilities, making it a cornerstone of ML lifecycle management and reproducible AI development.
• Stats: ⭐ 27,040 · 🍴 6,001 · last commit 2026-07-15 · Active
• Learning priority / difficulty: 4 / Intermediate
• Tags: mlops experiment-tracking model-registry evaluation ai-governance
• What to learn from it:
  • Tracking experiments, models, and evaluation runs in one registry.
  • Comparing model and prompt variants with MLflow's UI.
  • Enforcing governance and reproducibility in AI projects.
  • Integrating MLflow with existing data-science workflows.

comet-ml/opik

Debug, evaluate, and monitor LLM applications, RAG systems, and agentic workflows.

• Why it matters: Opik combines comprehensive tracing, automated evaluations, and production dashboards into a unified open-source platform built by the Comet team.
• Stats: ⭐ 20,608 · 🍴 1,607 · last commit 2026-07-15 · Active
• Learning priority / difficulty: 4 / Intermediate
• Tags: llm-observability evaluation tracing rag agentic-workflows
• What to learn from it:
  • Building an EDD dashboard with traces and scores.
  • Automating evals on every deployment.
  • Using Comet's ecosystem for ML and LLM evaluation continuity.
  • Bridging development and production observability.

iterative/dvc

Data versioning and ML experiments for reproducible AI development.

• Why it matters: DVC brings Git-like version control to datasets and models, enabling reproducible evaluation pipelines and experiment tracking for data-driven AI systems.
• Stats: ⭐ 15,750 · 🍴 1,313 · last commit 2026-07-14 · Active
• Learning priority / difficulty: 4 / Intermediate
• Tags: data-version-control reproducibility mlops experiments pipelines
• What to learn from it:
  • Versioning datasets and models alongside code.
  • Reproducing any historical evaluation run from a Git commit.
  • Building data pipelines that feed into eval workflows.
  • Why reproducibility is a prerequisite for EDD.

arize-ai/phoenix

AI observability and evaluation platform for LLM and agent applications.

• Why it matters: Phoenix provides open-source tools for tracing, evaluating, and debugging AI applications, with strong integrations for RAG and agent workflows.
• Stats: ⭐ 10,571 · 🍴 986 · last commit 2026-07-15 · Active
• Learning priority / difficulty: 4 / Intermediate
• Tags: llm-observability evaluation tracing rag agents
• What to learn from it:
  • Combining observability and evaluation in the same platform.
  • Running experiments that compare app versions using real traces.
  • Debugging why an agent failed by inspecting its trace.
  • Exporting evaluation datasets from production traces.

traceloop/openllmetry

Open-source observability for GenAI and LLM applications based on OpenTelemetry.

• Why it matters: OpenLLMetry standardizes LLM instrumentation through OpenTelemetry, making it easy to collect traces and evaluation signals across providers.
• Stats: ⭐ 7,302 · 🍴 1,018 · last commit 2026-07-13 · Active
• Learning priority / difficulty: 4 / Intermediate
• Tags: observability opentelemetry llm-tracing monitoring genai
• What to learn from it:
  • Instrumenting LLM apps with OpenTelemetry standards.
  • Collecting traces once and routing them to multiple backends.
  • Adding eval signals to standardized telemetry.
  • Avoiding vendor lock-in in your observability stack.

helicone/helicone

Open-source LLM observability platform with one-line integration for monitoring and evaluation.

• Why it matters: Helicone offers a lightweight, open-source observability layer with built-in prompt management, cost tracking, and evaluation features for LLM applications.
• Stats: ⭐ 5,948 · 🍴 627 · last commit 2026-07-05 · Active
• Learning priority / difficulty: 4 / Beginner
• Tags: llm-observability monitoring prompt-management evaluation cost-tracking
• What to learn from it:
  • Adding observability to an LLM app with minimal code changes.
  • Tracking cost, latency, and quality metrics together.
  • Using prompt management features to version and evaluate prompts.
  • When a lightweight EDD entry point is the right choice.

Evaluation Stack at a Glance

Repo	Sub-discipline(s)	⭐ Stars	Language	License	Last Commit	Activity	Difficulty	Ranking	Evals Prompts?	Evals Agents?	Trajectory?	CI/CD?
langfuse/langfuse	Prompt, Agent, EDD	31,197	TypeScript	NOASSERTION	2026-07-15	Active	Intermediate	Must Learn	✅	✅	✅	✅
promptfoo/promptfoo	Prompt, Agent, EDD	23,284	TypeScript	MIT	2026-07-15	Active	Beginner/Intermediate	Must Learn	✅	✅	✅	✅
openai/evals	Prompt, Agent	18,922	Python	NOASSERTION	2026-04-14	Moderate	Intermediate	Must Learn	✅	✅	⚠️	⚠️
mlflow/mlflow	Prompt, EDD	27,040	Python	Apache-2.0	2026-07-15	Active	Intermediate	Strongly Recommended	✅	⚠️	⚠️	✅
comet-ml/opik	Prompt, Agent, EDD	20,608	Python	Apache-2.0	2026-07-15	Active	Intermediate	Must/Strong	✅	✅	✅	✅
confident-ai/deepeval	Prompt, Agent, EDD	16,869	Python	Apache-2.0	2026-07-14	Active	Intermediate	Must Learn	✅	✅	⚠️	✅
EleutherAI/lm-evaluation-harness	Agent, EDD	13,289	Python	MIT	2026-07-13	Active	Advanced	Must/Strong	⚠️	✅	⚠️	✅
arize-ai/phoenix	Prompt, Agent, EDD	10,571	Python	NOASSERTION	2026-07-15	Active	Intermediate/Advanced	Strongly Recommended	✅	✅	✅	✅
explodinggradients/ragas	Agent	14,850	Python	Apache-2.0	2026-02-24	Moderate	Intermediate	Strongly Recommended	⚠️	✅	⚠️	⚠️
SWE-bench/SWE-bench	Agent	5,427	Python	MIT	2026-04-01	Moderate	Advanced	Strongly Recommended	❌	✅	✅	⚠️
UKGovernmentBEIS/inspect_ai	Agent	2,352	Python	MIT	2026-07-15	Active	Advanced	Strongly Recommended	⚠️	✅	✅	✅
langchain-ai/langsmith-sdk	Agent	968	Python	MIT	2026-07-15	Active	Intermediate	Strongly Recommended	✅	✅	✅	✅
iterative/dvc	EDD	15,750	Python	Apache-2.0	2026-07-14	Active	Intermediate	Strongly Recommended	❌	❌	❌	✅
traceloop/openllmetry	EDD	7,302	Python	Apache-2.0	2026-07-13	Active	Intermediate	Strongly Recommended	❌	❌	✅	✅
helicone/helicone	EDD	5,948	TypeScript	Apache-2.0	2026-07-05	Active	Beginner	Strongly Recommended	✅	⚠️	⚠️	✅
agenta-ai/agenta	Prompt	4,295	TypeScript	NOASSERTION	2026-07-15	Active	Intermediate	Strongly Recommended	✅	⚠️	⚠️	⚠️
truera/trulens	Prompt	3,438	Python	MIT	2026-07-11	Active	Intermediate	Strongly Recommended	✅	⚠️	⚠️	⚠️
hegelai/prompttools	Prompt	3,041	Python	Apache-2.0	2026-02-11	Moderate	Beginner	Emerging	✅	❌	❌	❌

Legend: ✅ first-class, ⚠️ partial or possible, ❌ not the primary focus.

Recommended Learning Order

Start with the tools that give you fast feedback, then add observability and CI integration:

1. promptfoo — Run your first declarative prompt evals locally.
2. DeepEval — Add Python-native unit tests and metrics.
3. Langfuse or Opik — Centralize traces, scores, and datasets.
4. Phoenix or Helicone — Add observability and production monitoring.
5. Inspect AI or SWE-bench — Graduate to multi-turn and coding-agent evaluation.
6. DVC + MLflow — Version data/models and track experiments for full EDD rigor.

Build your first eval pipeline in a weekend

1. Initialize: npx promptfoo@latest init or pip install promptfoo and create promptfooconfig.yaml.
2. Write test cases: Add 10-20 representative inputs with expected behaviors (not just exact outputs).
3. Add assertions: Use deterministic checks (contains, regex, json-schema) first, then add one LLM-as-judge rubric for quality.
4. Run locally: promptfoo eval and inspect the result table.
5. Version control: Commit the config and test cases alongside your prompts.
6. Wire into CI: Add a GitHub Action step that runs promptfoo eval and fails on regressions.
7. Close the loop: Feed failing or interesting cases back into a dataset in Langfuse, Opik, or DeepEval for continuous improvement.

When to use which

Tool	Best for	Team size	Hosting	Trajectory eval	Standout feature
promptfoo	Prompt and agent regression testing, red teaming	Any	OSS/self-hosted	Yes via YAML	Best-in-class CI/CD gating
DeepEval	Python teams who want pytest-style LLM unit tests	Small to large	OSS	Limited	Built-in metrics + synthetic data
Langfuse	End-to-end LLMOps: traces, evals, prompt mgmt	Small to enterprise	OSS/self-hosted	Yes	Integrated self-hosted stack
Opik	Comet users and teams wanting open observability	Small to enterprise	OSS/Cloud	Yes	Fast-growing, strong tracing
OpenAI Evals	Standardized public benchmarks and registry contributions	Research/any	OSS	No	Canonical benchmark registry
Inspect AI	Safety-critical and multi-turn agent evals	Research/enterprise	OSS	Yes	Human-in-the-loop scorers
Ragas	RAG agent evaluation without ground truth	Small to large	OSS	Partial	Reference-free RAG metrics
Phoenix	Observability-first eval and trace analysis	Small to enterprise	OSS/Cloud	Yes	Embeddings and RAG diagnostics
Helicone	Lightweight one-line observability + eval entry	Solo to small	OSS/Cloud	Partial	Minimal integration friction
MLflow	ML lifecycle and experiment governance	Enterprise	OSS/Cloud	Partial	Mature registry and tracking
Braintrust (not in dataset)	Enterprise prompt/agent eval SaaS	Enterprise	SaaS	Yes	Strong eval-driven product
Weave (not in dataset)	Weights & Biias-style LLM experiment tracking	Small to large	SaaS	Partial	Tight model-training lineage

Related sections

• README Section 1 — Overview of the awesome-agent-engineering knowledge base.
• Benchmarks — Foundation-model and agent benchmarks that provide the ground truth many of these tools consume.
• Observability — The tracing and monitoring layer that supplies the data these evals run on.