Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,8 @@

# Bonus Unit 2: Observability and Evaluation of Agents

<Tip>
You can follow the code in <a href="https://colab.research.google.com/#fileId=https%3A//huggingface.co/agents-course/notebooks/blob/main/bonus-unit2/monitoring-and-evaluating-agents.ipynb" target="_blank">this notebook</a> that you can run using Google Colab.
</Tip>
> [!TIP]
> You can follow the code in <a href="https://colab.research.google.com/#fileId=https%3A//huggingface.co/agents-course/notebooks/blob/main/bonus-unit2/monitoring-and-evaluating-agents.ipynb" target="_blank">this notebook</a> that you can run using Google Colab.

In this notebook, we will learn how to **monitor the internal steps (traces) of our AI agent** and **evaluate its performance** using open-source observability tools.

Expand Down
9 changes: 4 additions & 5 deletions units/en/unit1/actions.mdx
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
# Actions: Enabling the Agent to Engage with Its Environment

<Tip>
In this section, we explore the concrete steps an AI agent takes to interact with its environment.

We’ll cover how actions are represented (using JSON or code), the importance of the stop and parse approach, and introduce different types of agents.
</Tip>
> [!TIP]
> In this section, we explore the concrete steps an AI agent takes to interact with its environment.
>
> We’ll cover how actions are represented (using JSON or code), the importance of the stop and parse approach, and introduce different types of agents.

Actions are the concrete steps an **AI agent takes to interact with its environment**.

Expand Down
22 changes: 8 additions & 14 deletions units/en/unit1/thoughts.mdx
Original file line number Diff line number Diff line change
@@ -1,13 +1,10 @@

# Thought: Internal Reasoning and the ReAct Approach

<Tip>

In this section, we dive into the inner workings of an AI agent—its ability to reason and plan. We’ll explore how the agent leverages its internal dialogue to analyze information, break down complex problems into manageable steps, and decide what action to take next.

Additionally, we introduce the ReAct approach, a prompting technique that encourages the model to think “step by step” before acting.

</Tip>
> [!TIP]
> In this section, we dive into the inner workings of an AI agent—its ability to reason and plan. We’ll explore how the agent leverages its internal dialogue to analyze information, break down complex problems into manageable steps, and decide what action to take next.
>
> Additionally, we introduce the ReAct approach, a prompting technique that encourages the model to think “step by step” before acting.

Thoughts represent the **Agent's internal reasoning and planning processes** to solve the task.

Expand Down Expand Up @@ -85,10 +82,7 @@ Action: Finish["It's 18°C and cloudy in Paris."]
| External tools | ❌ No | ✅ Yes (Actions + Observations) |
| Best suited for | Logic, math, internal tasks | Info-seeking, dynamic multi-step tasks |

<Tip>

Recent models like **Deepseek R1** or **OpenAI’s o1** were fine-tuned to *think before answering*. They use structured tokens like `<think>` and `</think>` to explicitly separate the reasoning phase from the final answer.

Unlike ReAct or CoT — which are prompting strategies — this is a **training-level technique**, where the model learns to think via examples.

</Tip>
> [!TIP]
> Recent models like **Deepseek R1** or **OpenAI’s o1** were fine-tuned to *think before answering*. They use structured tokens like `<think>` and `</think>` to explicitly separate the reasoning phase from the final answer.
>
> Unlike ReAct or CoT — which are prompting strategies — this is a **training-level technique**, where the model learns to think via examples.
5 changes: 2 additions & 3 deletions units/en/unit1/what-are-agents.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -83,9 +83,8 @@ The most common AI model found in Agents is an LLM (Large Language Model), which

Well known examples are **GPT4** from **OpenAI**, **LLama** from **Meta**, **Gemini** from **Google**, etc. These models have been trained on a vast amount of text and are able to generalize well. We will learn more about LLMs in the [next section](what-are-llms).

<Tip>
It's also possible to use models that accept other inputs as the Agent's core model. For example, a Vision Language Model (VLM), which is like an LLM but also understands images as input. We'll focus on LLMs for now and will discuss other options later.
</Tip>
> [!TIP]
> It's also possible to use models that accept other inputs as the Agent's core model. For example, a Vision Language Model (VLM), which is like an LLM but also understands images as input. We'll focus on LLMs for now and will discuss other options later.

## How does an AI take action on its environment?

Expand Down
7 changes: 2 additions & 5 deletions units/en/unit1/what-are-llms.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -118,11 +118,8 @@ The table below illustrates the diversity of special tokens.
</tbody>
</table>

<Tip>

We do not expect you to memorize these special tokens, but it is important to appreciate their diversity and the role they play in the text generation of LLMs. If you want to know more about special tokens, you can check out the configuration of the model in its Hub repository. For example, you can find the special tokens of the SmolLM2 model in its <a href="https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct/blob/main/tokenizer_config.json">tokenizer_config.json</a>.

</Tip>
> [!TIP]
> We do not expect you to memorize these special tokens, but it is important to appreciate their diversity and the role they play in the text generation of LLMs. If you want to know more about special tokens, you can check out the configuration of the model in its Hub repository. For example, you can find the special tokens of the SmolLM2 model in its <a href="https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct/blob/main/tokenizer_config.json">tokenizer_config.json</a>.

## Understanding next token prediction.

Expand Down
5 changes: 2 additions & 3 deletions units/en/unit2/langgraph/document_analysis_agent.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,8 @@ The workflow we’ll build follows this structured schema:

![Butler's Document Analysis Workflow](https://huggingface.co/datasets/agents-course/course-images/resolve/main/en/unit2/LangGraph/alfred_flow.png)

<Tip>
You can follow the code in <a href="https://huggingface.co/agents-course/notebooks/blob/main/unit2/langgraph/agent.ipynb" target="_blank">this notebook</a> that you can run using Google Colab.
</Tip>
> [!TIP]
> You can follow the code in <a href="https://huggingface.co/agents-course/notebooks/blob/main/unit2/langgraph/agent.ipynb" target="_blank">this notebook</a> that you can run using Google Colab.

## Setting Up the environment

Expand Down
5 changes: 2 additions & 3 deletions units/en/unit2/langgraph/first_graph.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,8 @@ Now that we understand the building blocks, let's put them into practice by buil

This example demonstrates how to structure a workflow with LangGraph that involves LLM-based decision-making. While this can't be considered an Agent as no tool is involved, this section focuses more on learning the LangGraph framework than Agents.

<Tip>
You can follow the code in <a href="https://huggingface.co/agents-course/notebooks/blob/main/unit2/langgraph/mail_sorting.ipynb" target="_blank">this notebook</a> that you can run using Google Colab.
</Tip>
> [!TIP]
> You can follow the code in <a href="https://huggingface.co/agents-course/notebooks/blob/main/unit2/langgraph/mail_sorting.ipynb" target="_blank">this notebook</a> that you can run using Google Colab.

## Our Workflow

Expand Down
5 changes: 2 additions & 3 deletions units/en/unit2/langgraph/introduction.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,8 @@ In this unit, you'll discover:
### 4️⃣ [Alfred, the document Analyst agent](./document_analysis_agent)
### 5️⃣ [Quiz](./quizz1)

<Tip warning={true}>
The examples in this section require access to a powerful LLM/VLM model. We ran them using the GPT-4o API because it has the best compatibility with langGraph.
</Tip>
> [!WARNING]
> The examples in this section require access to a powerful LLM/VLM model. We ran them using the GPT-4o API because it has the best compatibility with langGraph.

By the end of this unit, you'll be equipped to build robust, organized and production ready applications !

Expand Down
11 changes: 6 additions & 5 deletions units/en/unit2/llama-index/agents.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,13 @@ LlamaIndex supports **three main types of reasoning agents:**
2. `ReAct Agents` - These can work with any AI that does chat or text endpoint and deal with complex reasoning tasks.
3. `Advanced Custom Agents` - These use more complex methods to deal with more complex tasks and workflows.

<Tip>Find more information on advanced agents on <a href="https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/agent/workflow/base_agent.py">BaseWorkflowAgent</a></Tip>
> [!TIP]
> Find more information on advanced agents on <a href="https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/agent/workflow/base_agent.py">BaseWorkflowAgent</a>

## Initialising Agents

<Tip>
You can follow the code in <a href="https://huggingface.co/agents-course/notebooks/blob/main/unit2/llama-index/agents.ipynb" target="_blank">this notebook</a> that you can run using Google Colab.
</Tip>
> [!TIP]
> You can follow the code in <a href="https://huggingface.co/agents-course/notebooks/blob/main/unit2/llama-index/agents.ipynb" target="_blank">this notebook</a> that you can run using Google Colab.

To create an agent, we start by providing it with a **set of functions/tools that define its capabilities**.
Let's look at how to create an agent with some basic tools. As of this writing, the agent will automatically use the function calling API (if available), or a standard ReAct agent loop.
Expand Down Expand Up @@ -155,6 +155,7 @@ agent = AgentWorkflow(
response = await agent.run(user_msg="Can you add 5 and 3?")
```

<Tip>Haven't learned enough yet? There is a lot more to discover about agents and tools in LlamaIndex within the <a href="https://docs.llamaindex.ai/en/stable/examples/agent/agent_workflow_basic/">AgentWorkflow Basic Introduction</a> or the <a href="https://docs.llamaindex.ai/en/stable/understanding/agent/">Agent Learning Guide</a>, where you can read more about streaming, context serialization, and human-in-the-loop!</Tip>
> [!TIP]
> Haven't learned enough yet? There is a lot more to discover about agents and tools in LlamaIndex within the <a href="https://docs.llamaindex.ai/en/stable/examples/agent/agent_workflow_basic/">AgentWorkflow Basic Introduction</a> or the <a href="https://docs.llamaindex.ai/en/stable/understanding/agent/">Agent Learning Guide</a>, where you can read more about streaming, context serialization, and human-in-the-loop!

Now that we understand the basics of agents and tools in LlamaIndex, let's see how we can use LlamaIndex to **create configurable and manageable workflows!**
20 changes: 12 additions & 8 deletions units/en/unit2/llama-index/components.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,8 @@ Now, let's dive a bit deeper into the components and see how you can **combine c

## Creating a RAG pipeline using components

<Tip>
You can follow the code in <a href="https://huggingface.co/agents-course/notebooks/blob/main/unit2/llama-index/components.ipynb" target="_blank">this notebook</a> that you can run using Google Colab.
</Tip>
> [!TIP]
> You can follow the code in <a href="https://huggingface.co/agents-course/notebooks/blob/main/unit2/llama-index/components.ipynb" target="_blank">this notebook</a> that you can run using Google Colab.

There are five key stages within RAG, which in turn will be a part of most larger applications you build. These are:

Expand All @@ -50,7 +49,8 @@ There are three main ways to load data into LlamaIndex:
2. `LlamaParse`: LlamaParse, LlamaIndex's official tool for PDF parsing, available as a managed API.
3. `LlamaHub`: A registry of hundreds of data-loading libraries to ingest data from any source.

<Tip>Get familiar with <a href="https://docs.llamaindex.ai/en/stable/module_guides/loading/connector/">LlamaHub</a> loaders and <a href="https://github.com/run-llama/llama_cloud_services/blob/main/parse.md">LlamaParse</a> parser for more complex data sources.</Tip>
> [!TIP]
> Get familiar with <a href="https://docs.llamaindex.ai/en/stable/module_guides/loading/connector/">LlamaHub</a> loaders and <a href="https://github.com/run-llama/llama_cloud_services/blob/main/parse.md">LlamaParse</a> parser for more complex data sources.

**The simplest way to load data is with `SimpleDirectoryReader`.**
This versatile component can load various file types from a folder and convert them into `Document` objects that LlamaIndex can work with.
Expand Down Expand Up @@ -124,7 +124,8 @@ pipeline = IngestionPipeline(
)
```

<Tip>An overview of the different vector stores can be found in the <a href="https://docs.llamaindex.ai/en/stable/module_guides/storing/vector_stores/">LlamaIndex documentation</a>.</Tip>
> [!TIP]
> An overview of the different vector stores can be found in the <a href="https://docs.llamaindex.ai/en/stable/module_guides/storing/vector_stores/">LlamaIndex documentation</a>.


This is where vector embeddings come in - by embedding both the query and nodes in the same vector space, we can find relevant matches.
Expand Down Expand Up @@ -176,7 +177,8 @@ Once again, this is fully customisable but there are three main strategies that
- `compact` (default): similar to refining but concatenating the chunks beforehand, resulting in fewer LLM calls.
- `tree_summarize`: create a detailed answer by going through each retrieved text chunk and creating a tree structure of the answer.

<Tip>Take fine-grained control of your query workflows with the <a href="https://docs.llamaindex.ai/en/stable/module_guides/deploying/query_engine/usage_pattern/#low-level-composition-api">low-level composition API</a>. This API lets you customize and fine-tune every step of the query process to match your exact needs, which also pairs great with <a href="https://docs.llamaindex.ai/en/stable/module_guides/workflow/">Workflows</a> </Tip>
> [!TIP]
> Take fine-grained control of your query workflows with the <a href="https://docs.llamaindex.ai/en/stable/module_guides/deploying/query_engine/usage_pattern/#low-level-composition-api">low-level composition API</a>. This API lets you customize and fine-tune every step of the query process to match your exact needs, which also pairs great with <a href="https://docs.llamaindex.ai/en/stable/module_guides/workflow/">Workflows</a>

The language model won't always perform in predictable ways, so we can't be sure that the answer we get is always correct. We can deal with this by **evaluating the quality of the answer**.

Expand All @@ -190,7 +192,8 @@ Let's look at the three main evaluators available:
- `AnswerRelevancyEvaluator`: Evaluate the relevance of the answer by checking if the answer is relevant to the question.
- `CorrectnessEvaluator`: Evaluate the correctness of the answer by checking if the answer is correct.

<Tip>Want to learn more about agent observability and evaluation? Continue your journey with the <a href="https://huggingface.co/learn/agents-course/bonus-unit2/introduction">Bonus Unit 2</a>.</Tip>
> [!TIP]
> Want to learn more about agent observability and evaluation? Continue your journey with the <a href="https://huggingface.co/learn/agents-course/bonus-unit2/introduction">Bonus Unit 2</a>.

```python
from llama_index.core.evaluation import FaithfulnessEvaluator
Expand Down Expand Up @@ -238,6 +241,7 @@ llama_index.core.set_global_handler(
)
```

<Tip>Want to learn more about components and how to use them? Continue your journey with the <a href="https://docs.llamaindex.ai/en/stable/module_guides/">Components Guides</a> or the <a href="https://docs.llamaindex.ai/en/stable/understanding/rag/">Guide on RAG</a>.</Tip>
> [!TIP]
> Want to learn more about components and how to use them? Continue your journey with the <a href="https://docs.llamaindex.ai/en/stable/module_guides/">Components Guides</a> or the <a href="https://docs.llamaindex.ai/en/stable/understanding/rag/">Guide on RAG</a>.

We have seen how to use components to create a `QueryEngine`. Now, let's see how we can **use the `QueryEngine` as a tool for an agent!**
11 changes: 6 additions & 5 deletions units/en/unit2/llama-index/tools.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,8 @@ We will go over each of them in more detail below.

## Creating a FunctionTool

<Tip>
You can follow the code in <a href="https://huggingface.co/agents-course/notebooks/blob/main/unit2/llama-index/tools.ipynb" target="_blank">this notebook</a> that you can run using Google Colab.
</Tip>
> [!TIP]
> You can follow the code in <a href="https://huggingface.co/agents-course/notebooks/blob/main/unit2/llama-index/tools.ipynb" target="_blank">this notebook</a> that you can run using Google Colab.

A FunctionTool provides a simple way to wrap any Python function and make it available to an agent.
You can pass either a synchronous or asynchronous function to the tool, along with optional `name` and `description` parameters.
Expand All @@ -41,7 +40,8 @@ tool = FunctionTool.from_defaults(
tool.call("New York")
```

<Tip>When using an agent or LLM with function calling, the tool selected (and the arguments written for that tool) rely strongly on the tool name and description of the purpose and arguments of the tool. Learn more about function calling in the <a href="https://docs.llamaindex.ai/en/stable/examples/workflow/function_calling_agent/">Function Calling Guide</a>.</Tip>
> [!TIP]
> When using an agent or LLM with function calling, the tool selected (and the arguments written for that tool) rely strongly on the tool name and description of the purpose and arguments of the tool. Learn more about function calling in the <a href="https://docs.llamaindex.ai/en/stable/examples/workflow/function_calling_agent/">Function Calling Guide</a>.

## Creating a QueryEngineTool

Expand Down Expand Up @@ -136,6 +136,7 @@ Let's walk through our two main utility tools below.
1. `OnDemandToolLoader`: This tool turns any existing LlamaIndex data loader (BaseReader class) into a tool that an agent can use. The tool can be called with all the parameters needed to trigger `load_data` from the data loader, along with a natural language query string. During execution, we first load data from the data loader, index it (for instance with a vector store), and then query it 'on-demand'. All three of these steps happen in a single tool call.
2. `LoadAndSearchToolSpec`: The LoadAndSearchToolSpec takes in any existing Tool as input. As a tool spec, it implements `to_tool_list`, and when that function is called, two tools are returned: a loading tool and then a search tool. The load Tool execution would call the underlying Tool, and then index the output (by default with a vector index). The search Tool execution would take in a query string as input and call the underlying index.

<Tip>You can find toolspecs and utility tools on the <a href="https://llamahub.ai/">LlamaHub</a></Tip>
> [!TIP]
> You can find toolspecs and utility tools on the <a href="https://llamahub.ai/">LlamaHub</a>

Now that we understand the basics of agents and tools in LlamaIndex, let's see how we can **use LlamaIndex to create configurable and manageable workflows!**
8 changes: 4 additions & 4 deletions units/en/unit2/llama-index/workflows.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,8 @@ So, let's learn how to create a workflow ourselves!

## Creating Workflows

<Tip>
You can follow the code in <a href="https://huggingface.co/agents-course/notebooks/blob/main/unit2/llama-index/workflows.ipynb" target="_blank">this notebook</a> that you can run using Google Colab.
</Tip>
> [!TIP]
> You can follow the code in <a href="https://huggingface.co/agents-course/notebooks/blob/main/unit2/llama-index/workflows.ipynb" target="_blank">this notebook</a> that you can run using Google Colab.

### Basic Workflow Creation

Expand Down Expand Up @@ -167,7 +166,8 @@ async def query(self, ctx: Context, ev: StartEvent) -> StopEvent:

Great! Now you know how to create basic workflows in LlamaIndex!

<Tip>There are some more complex nuances to workflows, which you can learn about in <a href="https://docs.llamaindex.ai/en/stable/understanding/workflows/">the LlamaIndex documentation</a>.</Tip>
> [!TIP]
> There are some more complex nuances to workflows, which you can learn about in <a href="https://docs.llamaindex.ai/en/stable/understanding/workflows/">the LlamaIndex documentation</a>.

However, there is another way to create workflows, which relies on the `AgentWorkflow` class. Let's take a look at how we can use this to create a multi-agent workflow.

Expand Down
Loading