You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/docs/Integrations/Cleanlab/integrations-cleanlab.md
+85-82Lines changed: 85 additions & 82 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,139 +7,142 @@ Unlock trustworthy Agentic, RAG, and LLM pipelines with Cleanlab's evaluation an
7
7
8
8
[Cleanlab](https://www.cleanlab.ai/) adds automation and trust to every data point going in and every prediction coming out of AI and RAG solutions.
9
9
10
-
This Langflow integration provides 3 modular components that assess and improve the **trustworthiness** of any LLM or RAG pipeline output, enabling critical oversight for safety-sensitive, enterprise, and production GenAI applications.
10
+
This Langflow integration provides three Langflow components that assess and improve the trustworthiness of any LLM or RAG pipeline output.
11
11
12
-
Use this bundle to:
13
-
- Quantify trustworthiness of ANY LLM response with a **0-1 score**
14
-
- Explain why a response may be good or bad
15
-
- Evaluate **context sufficiency**, **groundedness**, **helpfulness**, and **query clarity** with quantitative scores (for RAG/Agentic pipelines with context)
16
-
- Remediate low-trust responses with warnings or fallback answers
12
+
Use the components in this bundle to quantify the trustworthiness of any LLM response with a score between `0` and `1`, and explain why a response may be good or bad. For RAG/Agentic pipelines with context, you can evaluate context sufficiency, groundedness, helpfulness, and query clarity with quantitative scores. Additionally, you can remediate low-trust responses with warnings or fallback answers.
17
13
18
14
## Prerequisites
19
15
20
-
Before using these components, you'll need:
16
+
-[A Cleanlab API key](https://tlm.cleanlab.ai/)
21
17
22
-
- A [Cleanlab API key](https://tlm.cleanlab.ai/)
18
+
## CleanlabEvaluator
23
19
20
+
This component evaluates and explains the trustworthiness of a prompt and response pair using Cleanlab. For more information on how the score works, see the [Cleanlab documentation](https://help.cleanlab.ai/tlm/).
24
21
25
-
## Components
22
+
<details>
23
+
<summary>Parameters</summary>
26
24
27
-
### `CleanlabEvaluator`
25
+
**Inputs**
28
26
29
-
**Purpose:** Evaluate and explain the trustworthiness of a prompt + response pair using Cleanlab. More details on how the score works [here](https://help.cleanlab.ai/tlm/).
| system_prompt | Message | The system message prepended to the prompt. Optional. |
30
+
| prompt | Message | The user-facing input to the LLM. |
31
+
| response | Message | The model's response to evaluate. |
32
+
| cleanlab_api_key | Secret | Your Cleanlab API key. |
33
+
| cleanlab_evaluation_model | Dropdown | Evaluation model used by Cleanlab, such as GPT-4 or Claude. This does not need to be the same model that generated the response. |
34
+
| quality_preset | Dropdown | Tradeoff between evaluation speed and accuracy. |
| system_prompt | Message | (Optional) System message prepended to the prompt |
36
-
| prompt | Message | The user-facing input to the LLM |
37
-
| response | Message | OpenAI's, Claude, etc. model's response to evaluate |
38
-
| cleanlab_api_key | Secret | Your Cleanlab API key |
39
-
| cleanlab_evaluation_model | Dropdown | Evaluation model used by Cleanlab (GPT-4, Claude, etc.) This does not need to be the same model that generated the response. |
40
-
| quality_preset | Dropdown | Tradeoff between evaluation speed and accuracy |
| explanation | Message | Explanation of the trust score |
48
-
| response | Message | Returns the original response for easy chaining to `CleanlabRemediator` component |
46
+
## CleanlabRemediator
49
47
50
-
---
48
+
This component uses the trust score from the [CleanlabEvaluator](#cleanlabevaluator) component to determine whether to show, warn about, or replace an LLM response. This component has configurables for the score threshold, warning text, and fallback message that you can customize as needed.
51
49
52
-
### `CleanlabRemediator`
50
+
<details>
51
+
<summary>Parameters</summary>
53
52
54
-
**Purpose:** Use the trust score from the `CleanlabEvaluator` component to determine whether to show, warn about, or replace an LLM response. This component has configurables for the score threshold, warning text, and fallback message which you can customize as needed.
| remediated_response | Message | Final message shown to user after remediation logic |
71
+
</details>
73
72
73
+
## CleanlabRAGEvaluator
74
74
75
-
See example outputs below!
75
+
This component evaluates RAG and LLM pipeline outputs for trustworthiness, context sufficiency, response groundedness, helpfulness, and query ease. Learn more about Cleanlab's evaluation metrics [here](https://help.cleanlab.ai/tlm/use-cases/tlm_rag/).
76
76
77
-
---
77
+
Additionally, use the [CleanlabRemediator](#cleanlabremediator) component with this component to remediate low-trust responses coming from the RAG pipeline.
78
78
79
-
### `CleanlabRAGEvaluator`
79
+
<details>
80
+
<summary>Parameters</summary>
80
81
81
-
**Purpose:** Comprehensively evaluate RAG and LLM pipeline outputs by analyzing the context, query, and response quality using Cleanlab. This component assesses trustworthiness, context sufficiency, response groundedness, helpfulness, and query ease. Learn more about Cleanlab's evaluation metrics [here](https://help.cleanlab.ai/tlm/use-cases/tlm_rag/). You can also use the `CleanlabRemediator` component with this one to remediate low-trust responses coming from the RAG pipeline.
| cleanlab_api_key | Secret | Your Cleanlab API key. |
87
+
| cleanlab_evaluation_model | Dropdown | Thevaluation model used by Cleanlab, such as GPT-4, or Claude. This does not need to be the same model that generated the response. |
88
+
| quality_preset | Dropdown | The tradeoff between evaluation speed and accuracy. |
89
+
| context | Message | The retrieved context from your RAG system. |
90
+
| query | Message | The original user query. |
91
+
| response | Message | The model's response based on the context and query. |
| cleanlab_api_key | Secret | Your Cleanlab API key |
88
-
| cleanlab_evaluation_model | Dropdown | Evaluation model used by Cleanlab (GPT-4, Claude, etc.) This does not need to be the same model that generated the response. |
89
-
| quality_preset | Dropdown | Tradeoff between evaluation speed and accuracy |
90
-
| context | Message | Retrieved context from your RAG system |
91
-
| query | Message | The original user query |
92
-
| response | Message | OpenAI's, Claude, etc. model's response based on the context and query |
| evaluation_summary | Message | Markdown summary of query, context, response, and evaluation results |
106
+
</details>
106
107
107
-
---
108
+
## Cleanlab component example flows
108
109
109
-
## Example Flows
110
+
The following example flows show how to use the **CleanlabEvaluator** and **CleanlabRemediator** components to evaluate and remediate responses from any LLM, and how to use the `CleanlabRAGEvaluator` component to evaluate RAG pipeline outputs.
110
111
111
-
The following example flows show how to use the `CleanlabEvaluator`and `CleanlabRemediator` components to evaluate and remediate responses from any LLM, and how to use the `CleanlabRAGEvaluator` component to evaluate RAG pipeline outputs.
112
+
### Evaluate and remediate responses from an LLM
112
113
113
-
### Evaluate and remediate responses from any LLM
114
+
:::tip
115
+
Optionally, [Download](./eval_and_remediate_cleanlab.json) the Evaluate and Remediate flow and follow along.
116
+
:::
114
117
115
-
[Download](./eval_and_remediate_cleanlab.json) the flow to follow along!
116
-
117
-
This flow evaluates and remediates the trustworthiness of a response from any LLM using the `CleanlabEvaluator` and `CleanlabRemediator` components.
118
+
This flow evaluates and remediates the trustworthiness of a response from any LLM using the **CleanlabEvaluator** and **CleanlabRemediator** components.
Simply connect the `Message` output from any LLM component (like OpenAI, Anthropic, or Google) to the `response` input of the `CleanlabEvaluator` component, along with connecting your prompt to its `prompt` input.
122
+
Connect the `Message` output from any LLM component to the `response` input of the **CleanlabEvaluator** component, and then connect the Prompt component to its `prompt` input.
122
123
123
-
That's it! The `CleanlabEvaluator` component will return a trust score and explanation which you can use however you'd like.
124
+
The **CleanlabEvaluator** component returns a trust score and explanation from the flow.
124
125
125
-
The `CleanlabRemediator` component uses this trust score and user configurable settings to determine whether to output the original response, warn about it, or replace it with a fallback answer.
126
+
The **CleanlabRemediator** component uses this trust score to determine whether to output the original response, warn about it, or replace it with a fallback answer.
126
127
127
-
The example below shows a response that was determined to be untrustworthy (score of .09) and flagged with a warning by the `CleanlabRemediator` component.
128
+
This example shows a response that was determined to be untrustworthy (a score of `.09`) and flagged with a warning by the **CleanlabRemediator** component.
The below flow is the `Vector Store RAG` example template, with the `CleanlabRAGEvaluator` component added to evaluate the context, query, and response. You can use the `CleanlabRAGEvaluator` with any flow that has a context, query, and response. Simply connect the `context`, `query`, and `response` outputs from any RAG pipeline to the `CleanlabRAGEvaluator` component.
138
+
This example flow includes the [Vector Store RAG](/vector-store-rag) template with the **CleanlabRAGEvaluator** component added to evaluate the flow's context, query, and response.
139
+
140
+
To use the **CleanlabRAGEvaluator** component in a flow, connect the `context`, `query`, and `response` outputs from any RAG pipeline to the **CleanlabRAGEvaluator** component.
138
141
139
142
140
143
141
-
Here is an example of the `Evaluation Summary` output from the `CleanlabRAGEvaluator` component.
144
+
Here is an example of the `Evaluation Summary` output from the **CleanlabRAGEvaluator** component.
142
145
143
146
144
147
145
-
Notice how the `Evaluation Summary` includes the query, context, response, and all the evaluation results! In this example, the `Context Sufficiency` and `Response Groundedness` scores are low (0.002) because the context doesn't contain information about the query and the response is not grounded in the context.
148
+
The `Evaluation Summary` includes the query, context, response, and all evaluation results. In this example, the `Context Sufficiency` and `Response Groundedness` scores are low (a score of `0.002`) because the context doesn't contain information about the query, and the response is not grounded in the context.
0 commit comments