Skip to content
BranchesTags

Latest commit

 

History

5 Commits

Folders and files

NameName
Last commit message
Last commit date
assets
assets
 
 
data
data
 
 
docs
docs
 
 
scripts
scripts
 
 
.gitignore
.gitignore
 
 
CONTRIBUTING.md
CONTRIBUTING.md
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 
README.zh-CN.md
README.zh-CN.md
 
 

Repository files navigation

Awesome Agent Engineering

Awesome Agent Engineering

A long-term, continuously maintained knowledge base for Agent Engineering — the design, evaluation, optimization, benchmarking, and deployment of AI agents.

🎯 This list is organized around the Evaluation Layer. Evaluation is the highest-leverage activity in agent engineering — it is the only layer that turns guesswork into measurable, iteratively-improving systems. It is deliberately placed first.

English · 中文

Stats are fetched live via gh api; each category holds ≤ 10 repos (tier-aware, by stars). Last updated 2026-07-16.

Acknowledgments

Built with the help of two communities that keep agent engineering practical, not just theoretical:

How to use this repo

  1. Start with the Evaluation Layer (Section 1) and its flagship depth page docs/evals.md.
  2. Then read the agent Frameworks (§2) and Harness (§3) — the runtime substrate.
  3. Then the surfaces: MCP / Tools / Browser / Coding (§9–12).
  4. Then Infra (§15), Observability (§7), Memory (§8) — the supporting layers.
  5. Deep dives live in docs/ — one page per domain, each with a Mermaid diagram and a when-to-use-which guide.

Who it's for: backend/ML engineers building agents, platform & SRE teams operating them, app builders, and researchers benchmarking them.

Table of Contents

Categories

Overviews

Deep-dive pages (docs/)

1. Evaluation Layer ★

Evaluation is the leverage point of agent engineering: regression tests catch prompt regressions before users do, A/B compares variants, trajectory/multi-turn scoring measures real agent behavior, and Evaluation-Driven Development (EDD) closes the loop into CI/CD-for-AI. This section merges Prompt Evals, Agent Eval, and EDD because they are one discipline. Depth: docs/evals.md.

1.1 Prompt Evaluation

Evaluating and comparing prompts — regression testing, A/B, version comparison, pass/fail.

Repo ⭐ Forks Last commit Activity Priority Difficulty Summary
langfuse/langfuse 31197 3289 2026-07-15 Active 5 Intermediate Open-source AI engineering platform: evals, observability, prompt mgmt.
promptfoo/promptfoo 23284 2088 2026-07-15 Active 5 Beginner Declarative prompt/agent/RAG testing and red-teaming CLI.
openai/evals 18922 3025 2026-04-14 Moderate 5 Intermediate Framework and open registry for evaluating LLMs and LLM systems.
mlflow/mlflow 27040 6001 2026-07-15 Active 4 Intermediate Open-source AI engineering platform for agents, LLMs, and ML models.
comet-ml/opik 20608 1607 2026-07-15 Active 4 Intermediate Debug, evaluate, and monitor LLM apps with tracing and dashboards.
confident-ai/deepeval 16869 1661 2026-07-14 Active 4 Intermediate The LLM Evaluation Framework.
arize-ai/phoenix 10571 986 2026-07-15 Active 4 Advanced AI observability and evaluation.
agenta-ai/agenta 4295 568 2026-07-15 Active 4 Intermediate Open-source LLMOps platform: prompt playground, management, evaluation.
truera/trulens 3438 310 2026-07-11 Active 3 Intermediate Evaluation and tracking for LLM experiments and AI agents.
hegelai/prompttools 3041 256 2026-02-11 Moderate 3 Beginner Open-source tools for prompt testing and experimentation.

1.2 Agent Evaluation

Evaluating full agents — tool-use, multi-turn, trajectories, planning.

Repo ⭐ Forks Last commit Activity Priority Difficulty Summary
promptfoo/promptfoo 23285 2088 2026-07-15 Active 5 Intermediate Open-source LLM prompt, agent, and red-teaming evaluation platform.
comet-ml/opik 20608 1607 2026-07-15 Active 5 Intermediate Debug, evaluate, and monitor LLM applications, RAG systems, and agentic workflows.
openai/evals 18922 3025 2026-04-14 Moderate 5 Intermediate OpenAI's framework and registry for evaluating LLMs and LLM systems.
confident-ai/deepeval 16869 1661 2026-07-14 Active 5 Intermediate Open-source LLM evaluation framework with built-in metrics and test suites.
explodinggradients/ragas 14850 1564 2026-02-24 Moderate 4 Intermediate Framework for evaluating RAG and LLM applications with reference-free metrics.
EleutherAI/lm-evaluation-harness 13289 3409 2026-07-13 Active 4 Advanced Framework for few-shot evaluation of language models on hundreds of benchmarks.
arize-ai/phoenix 10571 986 2026-07-15 Active 4 Intermediate AI observability and evaluation platform for LLM, agent, and computer-vision applications.
SWE-bench/SWE-bench 5427 922 2026-04-01 Moderate 4 Advanced Benchmark for evaluating LLMs on real-world GitHub issue resolution.
UKGovernmentBEIS/inspect_ai 2352 603 2026-07-15 Active 4 Advanced Framework for large language model evaluations from the UK AI Safety Institute.
langchain-ai/langsmith-sdk 968 260 2026-07-15 Active 4 Intermediate LangSmith client SDK for tracing, evaluating, and monitoring LangChain applications.

1.3 Evaluation-Driven Development (EDD)

Iterative improvement through automated evaluation loops — regression, experiment tracking, CI/CD-for-AI.

Repo ⭐ Forks Last commit Activity Priority Difficulty Summary
langfuse/langfuse 31200 3289 2026-07-15 Active 5 Intermediate Open-source AI engineering platform for LLM evals, observability, metrics, prompt management, and datasets.
promptfoo/promptfoo 23285 2088 2026-07-15 Active 5 Intermediate Test prompts, agents, and RAGs with red teaming, pentesting, and CI/CD integration.
confident-ai/deepeval 16868 1661 2026-07-14 Active 5 Intermediate The LLM Evaluation Framework with built-in metrics and CI/CD integration.
EleutherAI/lm-evaluation-harness 13289 3409 2026-07-13 Active 5 Advanced A framework for few-shot evaluation of language models.
mlflow/mlflow 27040 6001 2026-07-15 Active 4 Intermediate Open-source AI engineering platform for debugging, evaluating, monitoring, and optimizing AI applications.
comet-ml/opik 20608 1607 2026-07-15 Active 4 Intermediate Debug, evaluate, and monitor LLM applications, RAG systems, and agentic workflows.
iterative/dvc 15750 1313 2026-07-14 Active 4 Intermediate Data versioning and ML experiments for reproducible AI development.
arize-ai/phoenix 10571 986 2026-07-15 Active 4 Intermediate AI observability and evaluation platform for LLM and agent applications.
traceloop/openllmetry 7302 1018 2026-07-13 Active 4 Intermediate Open-source observability for GenAI and LLM applications based on OpenTelemetry.
helicone/helicone 5948 627 2026-07-05 Active 4 Beginner Open-source LLM observability platform with one-line integration for monitoring and evaluation.

Why the Evaluation Layer comes first

Agents fail silently: a prompt tweak can drop tool-call accuracy 20% with no visible error. The Evaluation Layer is the only subsystem that makes agent behavior measurable and iteratively improvable rather than anecdotal. Every other layer (framework, harness, memory, infra) is only as good as the feedback loop that grades it — so evals come first, both in this list and in any mature agent stack.

flowchart LR
    DS[(Datasets /<br/>Eval Cases)] --> RUN[Prompt /<br/>Agent Run]
    RUN --> TR[Trace Capture]
    TR --> AS[Assertions +<br/>LLM Judge]
    AS --> SC[Aggregated Score]
    SC --> RG{Regression<br/>Gate}
    SC --> AB{A/B Compare}
    SC --> CI{CI/CD<br/>for AI}
    RG -. ship .-> SHP[✅ Ship]
    RG -. regress .-> IT[🔁 Iterate]
    AB --> IT
    CI --> IT
    IT --> RUN
Loading

2. Agent Frameworks

Top pick: langflow-ai/langflow — Visual framework for building and deploying AI-powered agents and workflows.

Repo ⭐ Forks Last commit Activity Priority Difficulty Summary
langflow-ai/langflow 151902 9675 2026-07-15 Active 5 Beginner Visual framework for building and deploying AI-powered agents and workflows.
langchain-ai/langchain 141831 23563 2026-07-15 Active 5 Intermediate The agent engineering platform — tools, chains, and agents for LLM applications.
modelcontextprotocol/servers 88505 11224 2026-07-10 Active 5 Intermediate Reference servers and protocol examples for MCP.
All-Hands-AI/OpenHands 80858 10324 2026-07-15 Active 5 Advanced AI-driven software development agent platform.
microsoft/autogen 59748 8996 2026-04-15 Moderate 5 Advanced A programming framework for agentic AI.
crewAIInc/crewAI 55563 7837 2026-07-15 Active 5 Intermediate Framework for orchestrating role-playing, autonomous AI agents.
langchain-ai/langgraph 37349 6257 2026-07-15 Active 5 Intermediate Build resilient agents with graph-based state machines.
openai/openai-agents-python 27926 4329 2026-07-15 Active 5 Intermediate Lightweight, powerful framework for multi-agent workflows.
modelcontextprotocol/python-sdk 23617 3645 2026-07-15 Active 5 Intermediate Official Python SDK for Model Context Protocol.
run-llama/llama_index 50865 7754 2026-07-13 Active 4 Intermediate Leading document agent and OCR platform.

3. Agent Harness

Top pick: langchain-ai/langgraph — Graph-based execution harness for building stateful, multi-step agent workflows with retries, branching, and persistence.

Repo ⭐ Forks Last commit Activity Priority Difficulty Summary
langchain-ai/langgraph 37349 6257 2026-07-15 Active 5 Intermediate Graph-based execution harness for building stateful, multi-step agent workflows with retries, branching, and persistence.
openai/openai-agents-python 27926 4329 2026-07-15 Active 5 Beginner Lightweight framework for multi-agent workflows with tracing, handoffs, and guardrails.
temporalio/temporal 21655 1735 2026-07-15 Active 5 Advanced Durable execution platform for long-running workflows with automatic retries, state persistence, and failure recovery.
mem0ai/mem0 60891 7093 2026-07-15 Active 4 Intermediate Universal memory layer for AI agents, enabling long-term state and personalization across runs.
microsoft/autogen 59748 8996 2026-04-15 Active 4 Intermediate Programming framework for agentic AI with multi-agent conversation patterns and code execution.
crewAIInc/crewAI 55563 7837 2026-07-15 Active 4 Intermediate Framework for orchestrating role-playing autonomous AI agents into collaborative crews.
google/adk-python 20620 3695 2026-07-15 Active 4 Intermediate Google's open-source Python toolkit for building, evaluating, and deploying AI agents.
pydantic/pydantic-ai 18544 2366 2026-07-15 Active 4 Intermediate Type-safe agent framework with structured outputs, dependency injection, and runtime evaluation.
camel-ai/camel 17396 2000 2026-07-14 Active 4 Advanced Multi-agent framework focused on communicative agents and scaling laws of agent societies.
inngest/inngest 5604 326 2026-07-15 Active 4 Intermediate Workflow orchestration platform for stateful step functions and AI workflows on serverless, servers, or edge.

4. Prompt Engineering

Top pick: stanfordnlp/dsp — DSPy: system for programming language models by writing declarative modules.

Repo ⭐ Forks Last commit Activity Priority Difficulty Summary
stanfordnlp/dsp 36148 3102 2026-07-15 Active 5 Advanced DSPy: system for programming language models by writing declarative modules.
langfuse/langfuse 31197 3289 2026-07-15 Active 5 Intermediate Open-source AI engineering platform with prompt management, evaluations, and observability.
promptfoo/promptfoo 23284 2088 2026-07-15 Active 5 Beginner Declarative prompt/agent evaluation and red-teaming framework with CI/CD integration.
microsoft/guidance 21667 1193 2026-05-21 Active 5 Advanced Microsoft DSL for controlling LLM generation via a guidance language.
outlines-dev/outlines 14508 777 2026-07-13 Active 4 Advanced Fast structured generation and constrained decoding for LLMs.
jxnl/instructor 13535 1154 2026-07-13 Active 4 Intermediate Python library for structured outputs from LLMs using Pydantic.
microsoft/promptflow 11183 1111 2026-07-09 Active 4 Intermediate End-to-end LLM application development and lifecycle management tool.
microsoft/TypeChat 8673 415 2026-07-07 Active 4 Intermediate Type-driven natural language interfaces for typed LLM interactions.
BoundaryML/baml 8544 448 2026-07-15 Active 4 Advanced Programming language and runtime for building reliable LLM-powered applications.
pezzolabs/pezzo 3255 276 2026-03-31 Moderate 4 Intermediate Open-source LLMOps platform focused on prompt design, versioning, and collaboration.

5. LLM Benchmark

Top pick: EleutherAI/lm-evaluation-harness — Widely-used few-shot evaluation harness for language models.

Repo ⭐ Forks Last commit Activity Priority Difficulty Summary
EleutherAI/lm-evaluation-harness 13289 3409 2026-07-13 Active 5 Intermediate Widely-used few-shot evaluation harness for language models.
princeton-nlp/SWE-bench 5427 922 2026-04-01 Moderate 5 Advanced Benchmark for evaluating LLMs on resolving real-world GitHub issues.
openai/human-eval 3301 451 2025-01-17 Stale 5 Beginner Original benchmark and dataset for evaluating code generation from docstrings.
stanford-crfm/helm 2855 404 2026-07-01 Active 5 Intermediate HELM provides a multi-metric, scenario-based evaluation framework that covers accuracy, calibration, robustness, efficiency, bias, and toxicity.
aider-ai/aider 47399 4733 2026-05-22 Active 4 Advanced AI pair programming tool with extensive benchmark support for code editing.
lm-sys/FastChat 39490 4785 2026-05-01 Active 4 Intermediate Open platform for training, serving, and evaluating LLMs, including Chatbot Arena.
stanfordnlp/dspy 36148 3102 2026-07-15 Active 4 Intermediate Framework for programming language models with optimizers and evaluation primitives.
promptfoo/promptfoo 23284 2088 2026-07-15 Active 4 Beginner Open-source evaluation and red-teaming framework for LLM prompts, agents, and RAG.
openai/evals 18922 3025 2026-04-14 Moderate 4 Intermediate Open-source registry of benchmarks and an evaluation framework for LLMs and LLM systems.
OpenRLHF/OpenRLHF 9795 988 2026-07-14 Active 4 Advanced Scalable RLHF framework with evaluation utilities for aligned models.

6. Agent Benchmark

Top pick: princeton-nlp/SWE-agent — An autonomous agent that fixes GitHub issues end-to-end using any language model.

Repo ⭐ Forks Last commit Activity Priority Difficulty Summary
princeton-nlp/SWE-agent 19818 2167 2026-07-13 Active 5 Advanced An autonomous agent that fixes GitHub issues end-to-end using any language model.
ShishirPatil/gorilla 12948 1389 2026-04-13 Moderate 5 Advanced Training and evaluation framework for large language models on API and tool calling.
OpenBMB/ToolBench 5697 486 2025-05-21 Stale 5 Advanced Open platform for training, serving, and evaluating large language models for tool learning.
princeton-nlp/SWE-bench 5427 922 2026-04-01 Moderate 5 Advanced Benchmark that evaluates whether language models can resolve real-world GitHub issues.
xlang-ai/OSWorld 3013 494 2026-07-15 Active 5 Advanced Benchmark for multimodal agents solving open-ended tasks in real computer environments.
harbor-framework/terminal-bench 2454 557 2026-07-11 Active 5 Advanced A benchmark for LLMs on complex tasks in the terminal.
openai/simple-evals 4570 495 2026-04-22 Moderate 3 Beginner Lightweight evaluation framework and registry for LLMs and LLM systems.
THUDM/AgentBench 3575 270 2026-02-08 Moderate 4 Advanced Comprehensive benchmark to evaluate LLMs as agents across multiple environments.
sierra-research/tau-bench 1328 209 2026-03-18 Moderate 4 Intermediate Benchmark and code for tau-bench, focusing on tool-augmented task agents.
OSU-NLP-Group/Mind2Web 1015 123 2025-11-05 Moderate 4 Intermediate Dataset and benchmark for generalist web agents based on large language models.

7. Observability

Top pick: grafana/grafana — Open observability and data visualization platform.

Repo ⭐ Forks Last commit Activity Priority Difficulty Summary
grafana/grafana 75647 14361 2026-07-15 Active 5 Intermediate Open observability and data visualization platform.
prometheus/prometheus 65231 10753 2026-07-15 Active 5 Advanced Open-source monitoring system and time series database.
langfuse/langfuse 31197 3289 2026-07-15 Active 5 Intermediate Open-source AI engineering platform for LLM observability, evals and prompt management.
mlflow/mlflow 27040 6001 2026-07-15 Active 5 Intermediate Open-source AI/ML lifecycle and observability platform.
comet-ml/opik 20608 1607 2026-07-15 Active 5 Intermediate Open-source observability, evaluation and monitoring for LLM, RAG and agent workflows.
confident-ai/deepeval 16869 1661 2026-07-14 Active 4 Intermediate LLM evaluation framework.
explodinggradients/ragas 14850 1564 2026-02-24 Moderate 4 Intermediate Evaluation framework for RAG and LLM applications.
arize-ai/phoenix 10571 986 2026-07-15 Active 4 Intermediate Open-source AI observability and evaluation platform.
traceloop/openllmetry 7302 1018 2026-07-13 Active 4 Intermediate OpenTelemetry-based observability instrumentation for GenAI/LLM apps.
open-telemetry/opentelemetry-go 6465 1417 2026-07-15 Active 4 Intermediate Official OpenTelemetry Go API and SDK.

8. Memory Systems

Top pick: mem0ai/mem0 — Universal memory layer for AI agents with user/professional identity, adaptive personalization, and broad framework integrations.

Repo ⭐ Forks Last commit Activity Priority Difficulty Summary
mem0ai/mem0 60891 7093 2026-07-15 Active 5 Intermediate Universal memory layer for AI agents with user/professional identity, adaptive personalization, and broad framework integrations.
topoteretes/cognee 27933 2760 2026-07-15 Active 5 Intermediate Open-source self-hosted AI memory platform that builds a knowledge graph from conversations and documents for persistent agent memory.
letta-ai/letta 23804 2521 2026-07-03 Active 5 Advanced Stateful agent platform (formerly MemGPT) with advanced memory management, self-editing memory, and long-context reasoning.
getzep/graphiti 28750 2903 2026-07-15 Active 4 Intermediate Real-time knowledge graph building library designed for agent memory and RAG.
supermemoryai/supermemory 28387 2472 2026-07-14 Active 4 Intermediate Fast, scalable, locally-runnable memory and context engine with a memory API and web app.
rohitg00/agentmemory 25166 2079 2026-07-13 Active 4 Beginner Persistent memory for AI coding agents, ranked #1 on real-world benchmarks.
TencentCloud/TencentDB-Agent-Memory 8933 820 2026-07-14 Active 3 Intermediate Fully local long-term memory for AI agents using a 4-tier progressive pipeline with no external API dependencies.
getzep/zep 4754 640 2026-07-10 Active 4 Intermediate Long-term memory store and platform for LLM applications with entity extraction, summarization, and classification.
langchain-ai/langmem 1562 176 2026-07-15 Active 4 Intermediate LangChain library for adding long-term memory to agents with built-in memory management tools.
NirDiamant/Agent_Memory_Techniques 788 104 2026-07-14 Active 4 Beginner Educational collection of 30 runnable notebooks covering buffers, vector stores, knowledge graphs, episodic/semantic memory, and production patterns.

9. MCP Ecosystem

Top pick: modelcontextprotocol/servers — Official reference implementations of MCP servers maintained by Anthropic.

Repo ⭐ Forks Last commit Activity Priority Difficulty Summary
modelcontextprotocol/servers 88505 11224 2026-07-10 Active 5 Intermediate Official reference implementations of MCP servers maintained by Anthropic.
jlowin/fastmcp 26217 2133 2026-07-14 Active 5 Beginner High-level, ergonomic Python framework for building MCP servers quickly.
modelcontextprotocol/python-sdk 23617 3645 2026-07-15 Active 5 Beginner Official Python SDK for building MCP servers and clients.
modelcontextprotocol/typescript-sdk 12860 1982 2026-07-15 Active 5 Beginner Official TypeScript SDK for building MCP servers and clients.
modelcontextprotocol/modelcontextprotocol 8599 1650 2026-07-15 Active 5 Intermediate Specification and documentation for the Model Context Protocol.
punkpeye/awesome-mcp-servers 90809 13019 2026-07-13 Active 4 Beginner Curated list of MCP servers and resources.
cline/cline 64683 6917 2026-07-15 Active 4 Intermediate Autonomous coding agent for VS Code with MCP support.
microsoft/playwright-mcp 35112 2922 2026-07-15 Active 4 Intermediate Microsoft's official browser automation MCP server using Playwright.
github/github-mcp-server 31463 4584 2026-07-15 Active 4 Intermediate GitHub's official MCP server for repository, issue, PR, and search operations.
tadata-org/fastapi_mcp 11944 952 2025-11-24 Moderate 4 Beginner Expose FastAPI endpoints as MCP tools.

10. Tool Calling

Top pick: denoland/deno — A modern runtime for JavaScript and TypeScript with built-in sandboxing, permissions, and secure code execution.

Repo ⭐ Forks Last commit Activity Priority Difficulty Summary
denoland/deno 107811 6283 2026-07-15 Active 5 Intermediate A modern runtime for JavaScript and TypeScript with built-in sandboxing, permissions, and secure code execution.
composiodev/composio 29243 4661 2026-07-15 Active 5 Intermediate Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench for AI agents.
openai/openai-agents-python 27926 4329 2026-07-15 Active 5 Intermediate A lightweight, powerful framework for multi-agent workflows.
modelcontextprotocol/python-sdk 23617 3645 2026-07-15 Active 5 Intermediate The official Python SDK for Model Context Protocol servers and clients.
daytonaio/daytona 72256 5661 2026-07-09 Active 4 Intermediate Secure and elastic infrastructure for running AI-generated code.
openai/swarm 21798 2324 2026-04-15 Moderate 4 Beginner Educational framework exploring ergonomic, lightweight multi-agent orchestration.
pyodide/pyodide 14736 1032 2026-07-15 Active 4 Advanced Python distribution for the browser and Node.js based on WebAssembly.
e2b-dev/E2B 12984 965 2026-07-15 Active 4 Intermediate Open-source, secure environment with real-world tools for enterprise-grade agents.
modelcontextprotocol/specification 8599 1650 2026-07-15 Active 4 Intermediate Specification and documentation for the Model Context Protocol.
browserbase/mcp-server-browserbase 3406 363 2026-07-07 Active 4 Intermediate Allow LLMs to control a browser with Browserbase and Stagehand.

11. Browser Agents

Top pick: browser-use/browser-use — 🌐 Make websites accessible for AI agents. Automate tasks online with ease.

Repo ⭐ Forks Last commit Activity Priority Difficulty Summary
browser-use/browser-use 104843 11550 2026-07-15 Active 5 Intermediate 🌐 Make websites accessible for AI agents. Automate tasks online with ease.
microsoft/playwright-mcp 35112 2922 2026-07-15 Active 5 Intermediate Playwright MCP server
browserbase/stagehand 23516 1616 2026-07-14 Active 5 Intermediate The SDK For Browser Agents
Skyvern-AI/skyvern 22243 2084 2026-07-15 Active 4 Intermediate Automate browser based workflows with AI
trycua/cua 19706 1296 2026-07-15 Active 4 Advanced Open-source infrastructure for Computer-Use Agents. Sandboxes, SDKs, and benchmarks to train and evaluate AI agents that can control full desktops (macOS, Linux, Windows).
browser-use/web-ui 16197 2715 2026-05-15 Active 4 Beginner 🖥️ Run AI Agent in your browser.
web-infra-dev/midscene 14078 1072 2026-07-15 Active 4 Intermediate AI-powered, vision-driven UI automation for every platform.
nanobrowser/nanobrowser 13467 1413 2025-11-24 Moderate 4 Beginner Open-Source Chrome extension for AI-powered web automation. Run multi-agent workflows using your own LLM API key. Alternative to OpenAI Operator.
steel-dev/steel-browser 7339 948 2026-07-12 Active 4 Intermediate 🔥 Open Source Browser API for AI Agents & Apps. Steel Browser is a batteries-included browser sandbox that lets you automate the web without worrying about infrastructure.
microsoft/Webwright 5808 367 2026-06-03 Active 4 Advanced A simple SWE style browser agent framework that achieves SOTA results on long horizon web tasks.

12. Coding Agents

Top pick: langchain-ai/langchain — The dominant agent engineering platform and framework.

Repo ⭐ Forks Last commit Activity Priority Difficulty Summary
langchain-ai/langchain 141832 23563 2026-07-15 Active 5 Advanced The dominant agent engineering platform and framework.
anthropics/claude-code 137941 22246 2026-07-14 Active 5 Intermediate Anthropic's official agentic coding CLI.
openai/codex 98364 14663 2026-07-15 Active 5 Intermediate OpenAI's official lightweight terminal coding agent.
All-Hands-AI/OpenHands 80859 10325 2026-07-15 Active 5 Advanced Open-source platform for AI software engineering agents.
google-gemini/gemini-cli 106001 14263 2026-07-15 Active 4 Intermediate Google's open-source terminal agent for the Gemini ecosystem.
cline/cline 64683 6917 2026-07-15 Active 4 Intermediate Autonomous coding agent as VS Code extension / SDK / CLI.
block/goose 51235 5694 2026-07-15 Active 4 Intermediate Block's extensible open-source coding agent.
Aider-AI/aider 47400 4733 2026-05-22 Active 4 Intermediate Terminal pair-programmer that edits files with your existing code.
continuedev/continue 34889 5050 2026-07-15 Active 4 Intermediate Open-source autopilot for VS Code.
TabbyML/tabby 33701 1768 2026-06-30 Active 3 Intermediate Self-hosted AI coding assistant.

13. RAG

Top pick: langchain-ai/langchain — The leading Python framework for building LLM applications with robust RAG tooling, vector stores, and agentic chains.

Repo ⭐ Forks Last commit Activity Priority Difficulty Summary
langchain-ai/langchain 141832 23563 2026-07-15 Active 5 Intermediate The leading Python framework for building LLM applications with robust RAG tooling, vector stores, and agentic chains.
infiniflow/ragflow 85105 9935 2026-07-15 Active 5 Intermediate Open-source RAG engine that fuses deep-document understanding with agentic retrieval and a visual workflow builder.
run-llama/llama_index 50865 7754 2026-07-13 Active 5 Intermediate Document-agent and data-framework for RAG with rich connectors, indices, and agentic retrieval primitives.
HKUDS/LightRAG 37699 5304 2026-07-15 Active 5 Intermediate Lightweight, fast GraphRAG implementation that combines graph and vector retrieval for cost-efficient knowledge-base QA.
microsoft/graphrag 34443 3641 2026-07-15 Active 5 Advanced Microsoft's modular graph-based RAG system using community detection and global/local query modes.
milvus-io/milvus 45234 4121 2026-07-15 Active 4 Advanced Cloud-native, distributed vector database for massive-scale ANN and RAG deployments.
langchain-ai/langgraph 37349 6257 2026-07-15 Active 4 Intermediate Agent orchestration framework from LangChain for building resilient, stateful multi-agent and RAG systems.
qdrant/qdrant 33296 2490 2026-07-15 Active 4 Intermediate High-performance Rust vector database with hybrid search, filtering, and cloud-native scaling.
chroma-core/chroma 28795 2385 2026-07-14 Active 4 Intermediate Developer-friendly vector database focused on simplicity, embeddings, and local-first RAG workflows.
deepset-ai/haystack 25902 2923 2026-07-15 Active 4 Intermediate Production-ready AI orchestration framework for modular RAG, semantic search, and agent pipelines.

14. Workflow Engines

Top pick: n8n-io/n8n — Fair-code workflow automation platform with native AI capabilities and 400+ integrations.

Repo ⭐ Forks Last commit Activity Priority Difficulty Summary
n8n-io/n8n 196534 59331 2026-07-15 Active 5 Beginner Fair-code workflow automation platform with native AI capabilities and 400+ integrations.
langflow-ai/langflow 151902 9675 2026-07-15 Active 5 Beginner Visual tool for building and deploying AI-powered agents and workflows.
langgenius/dify 148916 23443 2026-07-15 Active 5 Intermediate Production-ready open-source platform for building agentic AI workflows.
apache/airflow 46125 17390 2026-07-15 Active 5 Intermediate Programmatic authoring, scheduling, and monitoring of workflows.
temporalio/temporal 21655 1735 2026-07-15 Active 5 Advanced Durable execution engine for long-running workflows in Go and other languages.
FlowiseAI/Flowise 54648 24718 2026-07-15 Active 4 Beginner Open-source visual builder for AI agents and LLM workflows.
kestra-io/kestra 27435 2820 2026-07-15 Active 4 Intermediate Event-driven orchestration and scheduling platform using YAML/blueprint definitions.
PrefectHQ/prefect 23390 2410 2026-07-15 Active 4 Intermediate Modern Python workflow orchestration framework for resilient data pipelines.
windmill-labs/windmill 17163 1028 2026-07-15 Active 4 Intermediate Open-source developer platform and fast workflow engine with built-in webhooks and UIs.
argoproj/argo-workflows 16826 3571 2026-07-15 Active 4 Advanced Cloud-native workflow engine for Kubernetes.

15. AI Infrastructure

Top pick: ollama/ollama — Local LLM runtime and model management CLI/API.

Repo ⭐ Forks Last commit Activity Priority Difficulty Summary
ollama/ollama 176156 16968 2026-07-15 Active 5 Beginner Local LLM runtime and model management CLI/API.
ggerganov/llama.cpp 120461 20600 2026-07-15 Active 5 Advanced High-performance LLM inference in C/C++ with broad quantization support.
vllm-project/vllm 86324 19442 2026-07-15 Active 5 Advanced High-throughput, memory-efficient LLM serving engine.
BerriAI/litellm 53671 9786 2026-07-15 Active 5 Intermediate OpenAI-compatible AI gateway and SDK for 100+ LLM APIs.
sgl-project/sglang 30335 7171 2026-07-15 Active 5 Advanced High-performance serving framework for LLMs and multimodal models.
mudler/LocalAI 47542 4237 2026-07-15 Active 4 Intermediate Open-source AI engine for running LLMs, vision, voice, and image models locally.
ray-project/ray 43250 7794 2026-07-15 Active 4 Advanced Distributed AI compute engine and runtime for ML workloads.
mlc-ai/mlc-llm 22949 2088 2026-07-13 Active 4 Advanced Universal LLM deployment engine with ML compilation.
kvcache-ai/ktransformers 17437 1352 2026-07-15 Active 4 Advanced Heterogeneous LLM inference and fine-tune optimization framework.
Portkey-AI/gateway 12429 1206 2026-05-25 Active 4 Intermediate Fast AI gateway with guardrails, routing to 1600+ models.

16. Agent Safety

Top pick: promptfoo/promptfoo — Red teaming, pentesting, and vulnerability scanning for LLMs, agents, and RAGs.

Repo ⭐ Forks Last commit Activity Priority Difficulty Summary
promptfoo/promptfoo 23285 2088 2026-07-15 Active 5 Intermediate Red teaming, pentesting, and vulnerability scanning for LLMs, agents, and RAGs.
NVIDIA/garak 8446 1092 2026-07-14 Active 5 Advanced The LLM vulnerability scanner.
guardrails-ai/guardrails 7149 650 2026-07-14 Active 5 Intermediate Adding guardrails to large language models.
NVIDIA/NeMo-Guardrails 6708 768 2026-07-15 Active 5 Intermediate Programmable guardrails for LLM-based conversational systems.
microsoft/PyRIT 4113 808 2026-07-15 Active 5 Advanced Python Risk Identification Tool for generative AI.
microsoft/presidio 10007 1202 2026-07-15 Active 4 Intermediate Open-source PII detection, redaction, masking, and anonymization framework.
ibm/adversarial-robustness-toolbox 6108 1325 2025-12-12 Moderate 4 Advanced Python library for machine learning security — evasion, poisoning, extraction, inference.
llm-attacks/llm-attacks 4739 630 2024-08-02 Stale 4 Advanced Universal and Transferable Attacks on Aligned Language Models.
protectai/llm-guard 3184 423 2026-07-08 Archived 4 Intermediate The Security Toolkit for LLM Interactions.
0din-ai/ai-scanner 572 91 2026-07-14 Active 3 Intermediate AI model safety scanner built on NVIDIA garak.

17. Agent Optimization

Top pick: langchain-ai/langchain — The agent engineering platform for building applications with LLMs.

Repo ⭐ Forks Last commit Activity Priority Difficulty Summary
langchain-ai/langchain 141832 23563 2026-07-15 Active 5 Intermediate The agent engineering platform for building applications with LLMs.
microsoft/autogen 59748 8996 2026-04-15 Moderate 5 Intermediate Programming framework for agentic AI with conversational agents and optimization primitives.
stanfordnlp/dspy 36148 3102 2026-07-15 Active 5 Intermediate Framework for programming—not prompting—language models via composable modules and optimizers.
crewAIInc/crewAI 55565 7837 2026-07-15 Active 4 Intermediate Framework for orchestrating role-playing, autonomous AI agents.
langfuse/langfuse 31200 3289 2026-07-15 Active 4 Intermediate Open-source AI engineering platform: LLM evals, observability, metrics, prompt management, playground, datasets.
huggingface/smolagents 28366 2788 2026-07-14 Active 4 Beginner Barebones library for agents that think in code.
openai/swarm 21798 2324 2026-04-15 Moderate 3 Beginner Educational framework exploring lightweight multi-agent orchestration.
openai/evals 18922 3025 2026-04-14 Moderate 4 Intermediate Framework for evaluating LLMs and LLM systems, plus an open-source benchmark registry.
camel-ai/camel 17397 2000 2026-07-14 Active 3 Advanced The first and the best multi-agent framework, focused on communicative agents and agent society scaling.
confident-ai/deepeval 16868 1661 2026-07-14 Active 4 Intermediate The LLM evaluation framework with built-in metrics for correctness, faithfulness, and more.

18. Research Collections

Top pick: dair-ai/Prompt-Engineering-Guide — Open-source guide, papers, and resources for prompt engineering, RAG, and agents.

Repo ⭐ Forks Last commit Activity Priority Difficulty Summary
dair-ai/Prompt-Engineering-Guide 76504 8386 2026-03-11 Moderate 5 Beginner Open-source guide, papers, and resources for prompt engineering, RAG, and agents.
e2b-dev/awesome-ai-agents 28751 3151 2026-07-09 Active 5 Beginner Curated collection of autonomous AI agents, frameworks, and resources.
WooooDyy/LLM-Agent-Paper-List 8166 493 2025-09-12 Moderate 5 Intermediate Paper list accompanying the SCIS survey on large-language-model-based agents.
ai-boost/awesome-prompts 8473 801 2026-07-15 Active 4 Beginner Curated prompts and prompt-engineering resources from top GPTs.
atfortes/Awesome-LLM-Reasoning 3650 212 2026-04-20 Active 4 Intermediate Curated papers on LLM reasoning from Chain-of-Thought to OpenAI o1 and DeepSeek-R1.
DEEP-PolyU/Awesome-GraphRAG 2528 218 2026-06-02 Active 4 Advanced Curated surveys, papers, benchmarks, and projects on graph-based RAG.
AGI-Edgerunners/LLM-Agents-Papers 2330 152 2025-07-12 Stale 4 Intermediate Curated list of papers on LLM-based agents with daily auto-updates.
hymie122/RAG-Survey 1787 123 2024-08-20 Stale 4 Intermediate Curated papers on RAG foundations, enhancements, and AIGC applications.
asinghcsu/AgenticRAG-Survey 1700 181 2025-10-20 Moderate 4 Advanced Survey of agentic RAG systems enhanced with LLM agents.
taichengguo/LLM_MultiAgents_Survey_Papers 1291 65 2026-07-12 Active 4 Advanced Paper list for the IJCAI 2024 survey on LLM-based multi-agent systems.

19. Awesome Lists

Top pick: sindresorhus/awesome — The canonical Awesome meta-list of lists across all topics.

Repo ⭐ Forks Last commit Activity Priority Difficulty Summary
sindresorhus/awesome 485175 35874 2026-06-30 Active 5 Beginner The canonical Awesome meta-list of lists across all topics.
f/awesome-chatgpt-prompts 165801 21442 2026-07-15 Active 5 Beginner Massive community-curated prompt collection for ChatGPT and other LLMs.
dair-ai/Prompt-Engineering-Guide 76504 8386 2026-03-11 Moderate 5 Intermediate Comprehensive prompt engineering, RAG, and AI agent guide.
openai/openai-cookbook 74696 12644 2026-07-15 Active 5 Intermediate Official OpenAI cookbook with task-specific examples and recipes.
e2b-dev/awesome-ai-agents 28751 3151 2026-07-09 Active 5 Intermediate Curated list of AI autonomous agents, frameworks, and tools.
Hannibal046/awesome-llm 27143 2634 2025-07-31 Moderate 5 Intermediate Curated list of Large Language Model resources and breakthroughs.
awesome-selfhosted/awesome-selfhosted 305624 14320 2026-07-14 Active 4 Intermediate Curated list of self-hosted software and alternatives.
mlabonne/llm-course 80924 9425 2026-02-05 Moderate 4 Advanced Course to get into Large Language Models with roadmaps and notebooks.
PlexPt/awesome-chatgpt-prompts-zh 60972 13550 2026-04-28 Active 4 Beginner Chinese-language ChatGPT prompt engineering guide.
ml-tooling/best-of-ml-python 23680 3142 2026-07-09 Active 4 Advanced Ranked list of awesome machine learning Python libraries, updated weekly.

Cross-cutting Comparison Tables

Evaluation Layer comparison

Repo Stars Sub-discipline Prompts Agents Trajectory CI/CD License
langfuse/langfuse 31200 All Yes Yes Yes NOASSERTION
mlflow/mlflow 27040 ? ? ? ? Apache-2.0
promptfoo/promptfoo 23285 All Yes Yes Yes MIT
comet-ml/opik 20608 All Yes Yes Yes Apache-2.0
openai/evals 18922 All Yes Partial Yes NOASSERTION
confident-ai/deepeval 16868 Agent Yes Yes Yes Apache-2.0
iterative/dvc 15750 ? ? ? ? Apache-2.0
explodinggradients/ragas 14850 ? ? ? ? Apache-2.0
EleutherAI/lm-evaluation-harness 13289 ? ? ? ? MIT
arize-ai/phoenix 10571 ? ? ? ? NOASSERTION
traceloop/openllmetry 7302 ? ? ? ? Apache-2.0
helicone/helicone 5948 ? ? ? ? Apache-2.0
SWE-bench/SWE-bench 5427 ? ? ? ? MIT
agenta-ai/agenta 4295 ? ? ? ? NOASSERTION
truera/trulens 3438 ? ? ? ? MIT
hegelai/prompttools 3041 ? ? ? ? Apache-2.0
UKGovernmentBEIS/inspect_ai 2352 Agent Yes Yes Yes MIT
langchain-ai/langsmith-sdk 968 ? ? ? ? MIT

Agent Frameworks comparison

Repo Stars Multi-agent Graph/state Prod-ready Language
langflow-ai/langflow 151902 ? ? ? ?
langchain-ai/langchain 141831 ? ? ? ?
modelcontextprotocol/servers 88505 ? ? ? ?
All-Hands-AI/OpenHands 80858 Yes Yes Yes Python
microsoft/autogen 59748 Yes Yes Yes Python
crewAIInc/crewAI 55563 Yes Partial Yes Python
run-llama/llama_index 50865 ? ? ? ?
langchain-ai/langgraph 37349 Yes Yes Yes Python
openai/openai-agents-python 27926 Yes No Yes Python
modelcontextprotocol/python-sdk 23617 ? ? ? ?

Coding Agents comparison

Repo Stars Autonomous Copilot Self-host Language
langchain-ai/langchain 141832 ? ? ? ?
anthropics/claude-code 137941 ? ? ? ?
google-gemini/gemini-cli 106001 Yes Yes No TypeScript
openai/codex 98364 Yes Yes No Rust
All-Hands-AI/OpenHands 80859 Yes No Yes Python
cline/cline 64683 Yes Yes No TypeScript
block/goose 51235 Yes Yes Yes Rust
Aider-AI/aider 47400 Yes Yes Yes Python
continuedev/continue 34889 No Yes Yes TypeScript
TabbyML/tabby 33701 ? ? ? ?

Agent Engineering Architecture Stack

The Evaluation Layer is the emphasized central layer — it grades every other layer.

flowchart TB
    subgraph INF[AI Infrastructure / AI 基础设施]
        LiteLLM[vLLM · LiteLLM · Ollama]
    end
    subgraph FW[Frameworks / 框架]
        LG[LangGraph · CrewAI · AutoGen]
    end
    subgraph HRN[Harness / Runtime 运行时]
        TP[Temporal · LangGraph runtime]
    end
    subgraph SUR[Surfaces / 能力面]
        MCP[MCP / Tools · Browser · Coding agents]
    end
    subgraph MEM[Memory / 记忆]
        LT[Letta · Mem0 · Zep]
    end
    subgraph EV[★ Evaluation Layer 评估层 ★]
        PF[promptfoo · OpenAI Evals · DeepEval · Inspect AI · Langfuse]
    end
    subgraph OBS[Observability / 可观测性]
        OB[Phoenix · OpenLIT · Helicone]
    end
    subgraph BNC[Benchmarks / 基准]
        SW[SWE-bench · WebArena · OSWorld · GAIA · τ-bench]
    end
    INF --> FW --> HRN --> SUR
    MEM <--> HRN
    EV <==> OBS
    HRN -. traces 轨迹 .-> EV
    EV -. scores 打分 .-> BNC
    style EV fill:#fff4d6,stroke:#b8860b,stroke-width:3px,color:#000
Loading

End-to-End Agent + Eval Flow

An explicit evaluation gate decides ship-vs-iterate on every change.

flowchart LR
    U[User 用户] --> RT[Router / Gateway 路由网关]
    RT --> AL[Agent Loop 智能体循环<br/>plan·act·observe]
    AL --> TL[Tool / MCP 工具]
    AL --> MM[Memory 记忆]
    AL --> TR[Trace 轨迹]
    TR --> EG{★ EVAL GATE 评估门控 ★<br/>assertions · judge · regression}
    EG -->|pass 通过| SHP[✅ Ship 发布]
    EG -->|fail / drift 失败/漂移| IT[🔁 Iterate prompt / agent 迭代]
    IT --> AL
    style EG fill:#fff4d6,stroke:#b8860b,stroke-width:3px,color:#000
Loading

Recommended Learning Order

Begin with evaluation fundamentals before frameworks, to internalize that evals come first.

  1. promptfoo/promptfoo — Write your first prompt regression suite + assertions (Beginner/初级)
  2. openai/evals — Understand the canonical eval task + scoring harness shape (Intermediate/中级)
  3. confident-ai/deepeval — Add LLM-judge metrics + multi-turn/trajectory eval (Intermediate/中级)
  4. UKGovernmentBEIS/inspect_ai — Composable agent evals + scorer composition (Advanced/高级)
  5. langfuse/langfuse — Trace + dataset + eval in one loop (EDD) (Intermediate/中级)
  6. langchain-ai/langgraph — Build a stateful agent on a graph runtime (Intermediate/中级)
  7. All-Hands-AI/OpenHands — See a full autonomous coding agent harness (Advanced/高级)
  8. modelcontextprotocol/servers — Connect tools via MCP (Beginner/初级)
  9. browser-use/browser-use — Drive the browser autonomously (Intermediate/中级)
  10. BerriAI/litellm — Route + gatekeep model calls (Beginner/初级)
  11. letta-ai/letta — Add long-term memory to an agent (Advanced/高级)
  12. EleutherAI/lm-evaluation-harness — Run standard LLM benchmarks (Intermediate/中级)
  13. princeton-nlp/SWE-bench — Measure a coding agent on real issues (Advanced/高级)
  14. stanfordnlp/dsp — Programmatic prompt/agent optimization (Advanced/高级)

Learning Roadmap by Role

Each role: the Evaluation Layer repo(s) they must learn first, then their stack.

Backend / ML Engineer / 后端·ML 工程师

  • Eval first: promptfoo → DeepEval → Inspect AI
  • Then: LangGraph → PydanticAI → LiteLLM → Mem0

Platform / SRE / 平台·SRE

  • Eval first: Langfuse (trace + eval + EDD in CI) → promptfoo CLI gate
  • Then: LiteLLM → Phoenix/OpenLIT → Temporal → vLLM/SGLang

App / Product Builder / 应用·产品开发者

  • Eval first: promptfoo (regression before each release) → Opik
  • Then: Mastra/LangGraph → MCP servers → Browser Use → Continue/Cline

Researcher / 研究者

  • Eval first: OpenAI Evals → Inspect AI → lm-evaluation-harness
  • Then: SWE-bench/WebArena/GAIA/τ-bench → DSPy → HELM

Yearly Update — 2026

What shifted this year: reasoning models became the default substrate for hard agent tasks; MCP standardized tool attachment across frameworks and harnesses; coding agents matured from copilots into autonomous harnesses (OpenHands, Codex, Cline); and agentic evals / EDD crystallized into a recognized discipline — trajectory scoring and CI-for-AI moved from experiment to baseline.

Maintenance checklist

  • Re-run python3 scripts/refresh_stats.py quarterly to re-stamp stars/last-commit/activity
  • Keep ≤ 10 repos per category (tier-aware: Must Learn → Strongly Recommended → Emerging → Historical, by stars)
  • Prune newly-archived repos, or demote them to the Historical tier
  • The Evaluation Layer is the most volatile — review it most often; new eval harnesses appear fast
  • Re-run python3 scripts/gen_readme.py after any data/*.json edit to regenerate this README
  • Regenerate the docs/ depth pages when a category's repo set changes materially

Contributing

See CONTRIBUTING.md. Short version: verify a repo with gh api before proposing it; never hand-edit stats in data/*.json; assign a ranking tier; keep ≤ 10 per category; regenerate README + the affected docs page.

License

Code is MIT-licensed; the curated list content (README + docs) is CC-BY-4.0. See LICENSE.

Stats fetched via gh api on 2026-07-16 · ≤ 10 repos per category · generated by scripts/gen_readme.py.

About

A long-term, opinionated knowledge base for Agent Engineering — 21 categories, ≤10 curated repos each (live gh-api stats), the Evaluation Layer first, comparison tables, Mermaid diagrams, and a learning roadmap. EN + 中文.

Topics

agent-engineeringai-agentsawesome-listllm-benchmarkllm-evaluationmcp

Resources

Readme
MIT license

Contributing

Contributing
Activity

Stars

6 stars

Watchers

0 watching

Forks

2 forks
Report repository

Releases

Packages

Contributors

Languages