👋 Welcome!
We're currently developing an AI assistant 🤖 that helps you complete tasks through a web browser. We need your detailed feedback to help develop a more helpful and accurate AI assistant.
🎯 Here's what you'll be doing:
You will see a user's request (prompt), along with two attempts to solve it (Response A and Response B). We need your expert eye to help us figure out which one does a better job through a detailed, side-by-side comparison of the Responses.
✅ How to Evaluate:
To make your evaluation as useful as possible, please move beyond a simple "gut feeling." Instead, use the specific criteria provided to conduct a fair and objective evaluation. This structured approach is the most reliable way for us to understand what makes a response truly successful.
🔧 Tools:
On the right-hand side of the UI, there are various tools you can make use of:
- ℹ️ Click "Show Instructions" to see detailed instructions any time.
- ✏️ Take Notes: These can help you compile effective reasonings behind your ratings.
- ⚖️ Justification Builder: You may use this to help when performing fact-checking.
- 💬 Feedback: We'd love to hear your thoughts on this task, so please share any feedback!
🤖 Purpose: We're developing an AI assistant that helps you complete tasks through a web browser and we need your help understanding which responses are better.
🕵️ Your Role: Be a detective. Use the checklist to find the specific evidence for why one response is better, and report your findings.
To build your case, you will gather evidence by evaluating each response on these four key areas:
- ✅ Task Completion: How completely did the AI address the entirety of the user's request?
- 🎯 Factual Accuracy: How accurate are any facts or information provided?
- ⚙️ AI Performance: How would you rate the skill, quality and efficiency of the AI?
- 🏆 Best Response: After weighing all the evidence, this is your final conclusion.
- 🚨 Reporting Roadblocks: You will also be asked to report any technical problems that blocked or interfered with your investigation.
Objective
Your task is to define the "Gold Standard" for how the AI agent should respond to a specific user prompt.
A well crafted response should:
- Provide sources whenever is reasonable. Sources may come in the form of a product card, link to a website, or an open tab (shown as a screenshot below the response).
- When users ask for a list of options, it should include a reasonably sized list. Use your own judgement to decide if the list of options is either too long or too short based on the phrasing of the user prompt.
- Adhere to all of the search constraints specified by the user. If someone asks for a "recommendation for an action movie from the 90s featuring Tom Cruise," that includes three constraints for the agent to follow: genre (action), time period (90s), and actor (Tom Cruise).
- Stay on topic. It should not include off-topic or extraneous information.
A good guideline establishes the criteria for a good response for a given user prompt. It is Objective, Binary, and Comprehensive.
- Objective: Avoid vague phrases like "The response should be helpful." Instead, say "The response must list 3 specific stores."
- Binary: A third party should be able to read your guideline and the AI response and answer "Yes" or "No" without debate.
- Comprehensive: If the user asked for a specific date, price, location, or format, your guidelines must mention it.
⭐ Use the guides provided in these instructions to help you.
💡 Top Tip! It can be helpful to take notes during your evaluation! You can use our handy note-taking feature.
You will need to refer to some of this work in later parts of the task, taking notes will be very helpful!
Break the user's prompt down into specific constraints. Look for:
- Location: Cities, neighborhoods, specific radiuses, or "near me."
- Time: Dates, times of day, "next Friday," "open now."
- Quantity: "Top 3," "under $50," "party of 4," "size 6."
- Format: Tables, lists, maps, email drafts, calendar events.
- Action: Research, booking (or not booking), adding to cart, navigating.
Establish Ground Truth:
If the user asks a factual question (e.g., "Who won the Super Bowl last year?"), you must know the answer.
- Do not write: "The response must answer the question correctly."
- Do write: "The response must identify the Kansas City Chiefs as the winner."
Define the "Proof of Work":
Browser agents are expected to provide evidence. Your guidelines should almost always require:
- Links: Valid URLs to the source, product, or reservation page.
- Maps: Embedded maps for location queries.
- Product Cards: Visual snapshots for shopping items, hotels, or flights.
Writing your guidelines:
Structure your guidelines as a checklist. For each element in the checklist, make it clear if it is a 'should' or 'should not' do for the agent response.
While not required you may find it useful to start each item with 'should' or 'should not'.
Considering the user prompt 'Find me the lowest priced iPhone from the Apple store' a good set of guidelines would be:
The response:
- Should identify the lowest priced iPhone available from the Apple store.
- Should identify the specific model of the lowest priced iPhone.
- Should identify the relevant specifications of the lowest priced iPhone (e.g. storage capacity, color).
- Should provide a link to the page with a purchase option for the iPhone.
- Should NOT include results from other stores.
Section to see rating examples:
| User Prompt | Model Response | AI Performance | Justification |
|---|---|---|---|
| Respond to Angela's email about dinner plans. | Here's a draft response to reply to Angela.��Since your calendar shows you are busy every evening next week except for Wednesday and Thursday, I offered the following available times:��Wednesday, Dec 10: after 6:30pm�Thursday, Dec 11: after 5:30pm��< Note to rater : an action card for Gmail is embedded here. Pretend the card is correctly inserted.> | Excellent | There is a full response; it is personalized to the user; it uses an action card |
| Respond to Angela's email about dinner plans. | I don't have access to your calendar. Can you tell me when you are available next week so that I can draft a response to Angela? | Unacceptable | The agent failed to access the user's calendar. |
| Look up the hotel I reserved for Rome (should be in my email) and recommend some good gelato shops within walking distance. | You're in a great spot at the Singer Palace Hotel! You are right in the heart of Rome's historic center, meaning some of the city's most famous gelaterias are just a short stroll away… �[Multiple gelato shop recommendations with links and details] | Excellent | Found the hotel reservation; responses are close to the hotel; provides links and a product card |
| Look up the hotel I reserved for Rome (should be in my email) and recommend some good gelato shops within walking distance. | Here are some of the best and most highly-rated gelato shops in Rome. I have displayed them on the map for you… �[List of gelato shops without hotel context] | Poor | Provides a nice set of links and a product card BUT they aren't personalized (it doesn't associate the responses to a hotel reservation) |
| Look up the hotel I reserved for Rome (should be in my email) and recommend some good gelato shops within walking distance. | I can't find a hotel reservation in Rome. Can you tell me where you'll be staying? | Unacceptable | Both failed to do the personalization and failed to provide anything generic either |
| How much is an iPhone 16 (unlocked) on Amazon? Do they have the color pink in stock? | Here are the details for the Apple iPhone 16 (Unlocked) on Amazon:��Price: $540.00 for the Renewed (Refurbished) model with 128GB of storage.��Color Availability: Yes, the Pink color is in stock.��There are currently only 3 left in stock for this specific option.��Please note that this listing is for a Renewed device, not a brand-new one. | Excellent | Provides a valid link; meets all constraints; additionally is clear that it's not brand new |
Your goal here is to determine how well the AI completed every part of the user's request. A successful response doesn't just get the main idea right; it follows all the specific rules and instructions.
Follow these three steps to build your case and make an accurate rating:
Before looking at the responses, become an expert on the user's request. A single missed detail can change the entire case!
- Identify the Main Goal: What is the user trying to achieve?
- List all Sub-requests: What are the specific constraints, rules, or secondary tasks mentioned? (e.g., specific location, date, price, website, etc.)
Use this template to help organize your thoughts:
User's Main Goal: (e.g., "Find and compare vacation rentals.")
Sub-requests:
(e.g., "Must be in Scottsdale")
(e.g., "Must be for a group of 4")
(e.g., "Must provide multiple options")
Now, use your "Case File" from Step 1 as a checklist. For each response (A and B), check if it met the Main Goal and every single Sub-request.
- Fully Complete (5/5): Main task and ALL conditions were completed
- Mostly Complete (4/5): Main task was completed, but some minor conditions were incomplete
- Partially Complete (3/5): Main task was completed but several important conditions were incomplete
- Minimally Complete (2/5): One minor condition was completed but the main task was incomplete
- Not at all Complete (1/5): Neither main task nor conditions were completed
Important: Do NOT penalize Task Completion for missing links/screenshots if they were not required by the prompt.
Use this template to help organize your thoughts:
Response A
- Main Goal Fulfilled? [Yes/No]
- Sub-request 1: [Yes/No]
- Sub-request 2: [Yes/No]
- Sub-request 3: [Yes/No]
- Final Rating: [Rating]
- Justification: [Brief explain how you came to your conclusion]
Response B
- Main Goal Fulfilled? [Yes/No]
- Sub-request 1: [Yes/No]
- Sub-request 2: [Yes/No]
- Sub-request 3: [Yes/No]
- Final Rating: [Rating]
- Justification: [Brief explain how you came to your conclusion]
Your justification is your final report. It's the most valuable part of your investigation because it explains the evidence behind your rating.
This step is super important to us! Use the information you gathered in the steps above to be specific.
Prompt: Find me a few VRBO or AirBnb options in Scottsdale for the week of Valentine's Day for a group of 4.
Case File:
- Main Goal: Find and compare vacation rentals.
- Sub-requests: Must be in Scottsdale, must be for a group of 4, must provide multiple ("a few") options.
Investigation Notes:
-
Response A met the main goal and all 3 sub-requests, earning a 5/5 (Fully Complete).
- Justification: The response successfully met the main goal and all three sub-requests. It provided three relevant options that matched every constraint in the prompt, making it a perfect and useful answer.
-
Response B met the main goal but failed all 3 sub-requests, earning a 2/5 (Minimally Complete).
- Justification: While it found a vacation rental (fulfilling the main goal at a basic level), it failed on all three critical sub-requests. The location was wrong (Phoenix instead of Scottsdale), the group size was incorrect (2 people instead of 4), and it only provided a single option.
Final Verdict: "I chose Response A as much better. The primary and deciding factor was Task Completion. Response A successfully addressed every part of the prompt, including all three critical sub-requests. In stark contrast, Response B failed on every single sub-request, making it completely irrelevant to the user's needs. This made Response A a useful and actionable answer, while Response B was unhelpful."
NOTE: You can use the handy Justification Builder found in the side toolbar to help structure your justification.
A claim is any statement in the response that can be checked for accuracy—it can be proven either true or false. You should only fact-check objective claims.
-
✅ Objective: A statement that can be proven true or false with evidence (facts, data, etc.)
- Example: "The Eiffel Tower is in Paris."
-
❌ Subjective: A statement of opinion, belief, or personal feeling. It cannot be proven true or false.
- Example: "The Eiffel Tower is beautiful."
Be thorough! Break down the response into individual, numbered claims.
- Extract ALL objective claims individually (no merging or clubbing!). Keep them atomic.
- Copy claims exactly as they appear in the model response. Do not edit, rephrase, or add any claims that are not explicitly provided.
- Put reasoning in the Factual Accuracy section, not in the claims list.
- Action-based claims (e.g., "Email was drafted", "Google Doc was created", "I checked the website", "Calendar event was scheduled") MUST ALWAYS be listed, as they are verifiable actions.
- Information inside docs: If the info is general/non-private (e.g., "The document contains a grocery list"), include it as a claim, but label it Cannot Confidently Assess (CCA) since it requires doc access to verify. If the info is private user data (e.g., personal emails, addresses, phone numbers, account details), do NOT include it as a claim and ignore it for FA evaluation.
Prompt: What is Haku's true identity in Spirited Away?
Response: Haku's true identity is a fascinating part of the Spirited Away story! In the film, Haku is a mysterious boy who works at the bathhouse and helps Chihiro navigate the spirit world. However, it's later revealed that Haku is actually the spirit of the Kohaku River. He was once a powerful river spirit, but after being injured by humans, he lost his memory and ended up working for Yubaba. When Chihiro incorrectly guesses his true identity, Haku is finally able to break free from Yubaba's control and return to his river.
Here's how we'd identify and label the claims:
Haku's true identity is a fascinating part of the Spirited Away story! (❌ Subjective)
In the film, Haku is a mysterious boy who works at the bathhouse and helps Chihiro navigate the spirit world. (✅ Objective)
Haku is actually the spirit of the Kohaku River. (✅ Objective)
He was once a powerful river spirit, but after being injured by humans, he lost his memory and ended up working for Yubaba. (✅ Objective)
When Chihiro incorrectly guesses his true identity, Haku is finally able to break free... (✅ Objective - and in this case, factually wrong!)
Use reliable sources to verify or refute each objective claim. Don't rely on just one source!
What are Reputable Sources?
- Authority: Is the source a known expert? (e.g., a scientific journal, a respected museum)
- Objectivity: Is the source fair and unbiased?
- Accuracy: Is the source known for being correct?
- Currency: Is the information up-to-date?
👍 Good Examples: Encyclopedias (like Britannica), academic journals, reputable news organizations (like the New York Times, BBC, Reuters), government websites, official company/organization websites.
👎 Less Reliable Examples: Wikipedia (good for starting, not for final proof), personal blogs, social media posts, websites with obvious bias.
Use this if you don't use the Justification Builder.
Objective Claims Identified:
[Claim 1 text...]
[Claim 2 text...]
[And so on...]
Claim 1: [Rating Given]
Justification: [Brief explanation of your reasoning, referencing the sources.]
Sources: [Source 1 URL] [Source 2 URL]
Claim 2: [Rating Given]
Justification: [Brief explanation of your reasoning, referencing the sources.]
Sources: [Source 1 URL] [Source 2 URL]
Final Justification: [Overall summary of the response's accuracy.]
Applies to: Add-to-cart actions, Email sent claims, Calendar event creation, Dynamic prices or availability, or any external action that cannot be verified without proof.
- ✅ Case 1: Valid Evidence Provided (Screenshot or Link)
- Example: "I have added the Hat to your cart" (with a valid screenshot of the cart).
- FA: Completely Accurate (5) | TC: Fully Complete (5)
- ⚖️ Case 2: Evidence Partially Verifies the Claim
- Example: Prompt asked to add to cart. Response shows a screenshot of the product page, but not the cart.
- FA: Cannot Confidently Assess | TC: Partially Complete (3)
- ❌ Case 3: No Evidence Provided (Only AI Claim)
- If the response claims completion but provides no screenshot/link.
- FA: Cannot Confidently Assess (for the unverifiable claim) | TC: 3–4 (Partially or Mostly Complete)
- Core Rule: If FA is rated as Cannot Confidently Assess for a critical claim, TC cannot be Fully Complete (5).
- 🚫 Case 4: Screenshot Is Irrelevant to the Task
- If the screenshot does not relate to the claimed action.
- FA: Cannot Confidently Assess | TC: Not at All Complete (1)
- Valid, accurate, or invalid links -> No need to add these as separate FA claims.
- Inaccurate links only (links that redirect to incorrect products/domains) -> Add as a claim and mark as "Inaccurate".
- Provide direct, credible URLs as sources for your justifications — no Google search result links!
This question is different from the others. We're not just looking at the final answer—we're judging the skill and intelligence the AI showed along the way. Think of yourself as a performance reviewer watching the AI work.
The key question is: Did the AI achieve the goal gracefully and efficiently, or did it stumble and get lucky?
When evaluating performance, keep an eye on these four behaviors:
- 🗺️ Navigation: Was the path logical and direct, or did it meander to irrelevant pages?
- 🖱️ Interaction: Did it click the correct buttons and links precisely, or did it struggle and make mistakes?
- ⏱️ Efficiency: Were there wasted steps, loops, or unnecessary actions?
- 🧹 Output Quality: Is the final scraped text clean and well-formatted, or messy and filled with junk?
Here's a breakdown of what each rating means, with concrete examples. Click on each rating to see the details.
(5) Excellent (Expert & Optimal)
- Ask yourself: Was this a flawless, expert-level performance?
- Look for: Direct navigation, precise clicks, no wasted steps, and a clean final output. The AI behaved exactly as a skilled human would.
(4) Good (Effective but with minor flaws)
- Ask yourself: Was it almost perfect, but with one small mistake?
- Look for: A solid and logical process that had one minor, observable flaw. For example, it took one extra, unnecessary click, or the final output had a tiny formatting error.
(3) Acceptable (Clumsy but Functional)
- Ask yourself: Did it get the job done, but in a clumsy or inefficient way? This is the "it works, but..." category.
- Look for: An inefficient path (e.g., going to the Help page before finding the Contact page), clumsy actions (e.g., clicking an ad before correcting itself), or a messy but still usable final output.
(2) Poor (Illogical & Flawed)
- Ask yourself: Was the process itself a failure, even if it got a tiny bit of the answer right?
- Look for: Illogical steps where the AI seems confused or lost. It might struggle repeatedly to find the right button, or it only succeeded by chance, not by skill.
(1) Unacceptable (Broken or Nonsensical)
- Ask yourself: Did the AI completely fail to perform the actions required?
- Look for: Getting stuck on a page, being caught in a loop, timing out, or being fundamentally incapable of interacting with the website correctly.
Your justification MUST be at least 50 words and clearly cover ALL four dimensions: Navigation, Interaction, Efficiency, and Output Quality. Use actual examples from the response, not generic statements. Do not penalize AI Performance for issues that belong to other sections (like FA or TC). Zero AI usage is permitted when writing your justifications.
Use this template to help structure your thoughts:
Response A: [Rating]
Justification: [Brief explanation of your reasoning]
Response B: [Rating]
Justification: [Brief explanation of your reasoning]
Best Response: (e.g., "Response A, significantly.")
🕵️ You've investigated both responses and collected your evidence by answering the detailed questions. Now it's time to make your final call and write your report (rationale).
First, review your scores for each response side-by-side to identify the key differences.
| Question | Your Score for A | Your Score for B |
|---|---|---|
| Task Completion | [e.g., 5] | [e.g., 3] |
| Factual Accuracy | [e.g., 5] | [e.g., 5] |
| AI Performance | [e.g., 4] | [e.g., 5] |
- 👑 Correctness is King: Task Completion and Factual Accuracy are almost always more important than AI Performance. A clumsy response that gets the right answer is better than a graceful response that gets the wrong answer.
Example: "Response A fully completed the task by adhering to all requests. In contrast, Response B failed on a key sub-request, making it significantly less useful."
- 👔 Performance is the Tie-Breaker: If both responses are correct and complete, use AI Performance to decide which is "slightly better." The more efficient and skillful agent wins. Note: If the winning response is only better by one point for the deciding criterion, then it is only slightly better. Do not force "About the Same" when a clear winner exists. Ensure TC + FA + AIP align with your final verdict. Do not choose a winner based on preference.
Example: "While both responses were fully accurate, Response A's skillful and efficient navigation made it a higher-quality response overall."
Your written justification is critical! It explains the evidence behind your decision. Use the template below to structure your thoughts and be specific.
Use this template in the justification box:
Response A Scores:
- Task Completion: [1-5 Rating]
- Factual Accuracy: [1-5 Rating]
- AI Performance: [1-5 Rating]
Response B Scores:
- Task Completion: [1-5 Rating]
- Factual Accuracy: [1-5 Rating]
- AI Performance: [1-5 Rating]
Final Verdict: [e.g., Response A is much better.]
Justification: [In 30+ words, write the reason for your final recommendation, referencing the scores and Golden Rules above. Explain the most important difference that led to your choice.]