feat(web-search): add domain filtering, user location, and structured search actions

[leseb]

Closes #4442

Summary

Implements domain filtering and user location for the web search tool, matching the OpenAI Responses API feature set, and adds structured WebSearchToolCall.action models to surface search metadata (sources, queries) in responses.

API changes

• Add filters (domain include/exclude lists) and user_location (country, city, region, timezone) fields to OpenAIResponseInputToolWebSearch
• Add WebSearchToolCall action models: search, open_page, find with source URL metadata

Provider changes

• Brave Search: domain filtering, user location, search context size, and fix result slicing to align with search_context_size count
• Bing Search: domain filtering, user location, search context size
• Tavily Search: domain filtering, search context size
• All search backends now return structured source metadata (url, title) via ToolInvocationResult.metadata

Responses layer

• Thread filters and user_location from the web search tool config through the tool executor to search backends
• Populate WebSearchToolCall.action with source URLs from search results

Test plan

• Unit test test_invoke_with_search_context_size_updates_result_limit verifies brave search result slicing fix
• uv run pytest tests/unit/providers/tool_runtime/test_brave_search.py -x --tb=short
• uv run pre-commit run --all-files
• Integration tests for gpt/azure responses need re-recorded fixtures (web search request shapes changed)

Test script

#!/usr/bin/env bash
set -euo pipefail

BASE_URL="${OGX_BASE_URL:-http://localhost:8321}"
MODEL="${OGX_MODEL:-openai/gpt-4o-mini}"

pass=0
fail=0

CHECKER_DIR=$(mktemp -d)
trap 'rm -rf "$CHECKER_DIR"' EXIT

cat > "$CHECKER_DIR/expect_completed.py" << 'PYEOF'
import sys, json
data = json.load(sys.stdin)
if "detail" in data:
    print(f"  ERROR: {data['detail']}")
    sys.exit(1)
for o in data.get("output", []):
    t = o.get("type")
    if t == "web_search_call":
        print(f"  web_search_call: status={o.get('status')}")
        action = o.get("action")
        if action:
            print(f"    action.type={action['type']}")
            sources = action.get("sources", [])
            print(f"    sources={len(sources)}")
            for s in sources[:3]:
                print(f"      - {s['url']}")
    elif t == "message":
        print(f"  message: {o.get('content', [{}])[0].get('text', '')[:150]}")
ok = any(o.get("type") == "web_search_call" and o.get("status") == "completed" for o in data.get("output", []))
print("  => PASS" if ok else "  => FAIL")
sys.exit(0 if ok else 1)
PYEOF

cat > "$CHECKER_DIR/expect_validation_error.py" << 'PYEOF'
import sys, json
data = json.load(sys.stdin)
err = data.get("detail", "") or data.get("error", {}).get("message", "")
if err:
    print(f"  error: {str(err)[:200]}")
    print("  => PASS (rejected as expected)")
    sys.exit(0)
else:
    print(f"  unexpected response: {str(data)[:200]}")
    print("  => FAIL (should have been rejected)")
    sys.exit(1)
PYEOF

run_test() {
    local name="$1"
    local body="$2"
    local checker="$3"

    echo "=== $name ==="
    curl -s "$BASE_URL/v1/responses" \
        -X POST -H "Content-Type: application/json" \
        -d "$body" | python3 "$CHECKER_DIR/$checker" \
        && pass=$((pass + 1)) || fail=$((fail + 1))
    echo ""
}

# --- Happy path tests ---

run_test "Test 1: Basic web search" \
    "{\"model\": \"$MODEL\", \"input\": \"What were the top news headlines today, May 26 2026?\", \"tools\": [{\"type\": \"web_search\"}]}" \
    expect_completed.py

run_test "Test 2: search_context_size=low" \
    "{\"model\": \"$MODEL\", \"input\": \"What is the current price of Bitcoin today?\", \"tools\": [{\"type\": \"web_search\", \"search_context_size\": \"low\"}]}" \
    expect_completed.py

run_test "Test 3: Domain filtering" \
    "{\"model\": \"$MODEL\", \"input\": \"What is new in Python 3.13?\", \"tools\": [{\"type\": \"web_search\", \"filters\": {\"allowed_domains\": [\"python.org\", \"docs.python.org\"]}}]}" \
    expect_completed.py

run_test "Test 4: User location" \
    "{\"model\": \"$MODEL\", \"input\": \"What are the local election results this week?\", \"tools\": [{\"type\": \"web_search\", \"user_location\": {\"type\": \"approximate\", \"country\": \"FR\", \"city\": \"Paris\"}}]}" \
    expect_completed.py

run_test "Test 5: All features combined" \
    "{\"model\": \"$MODEL\", \"input\": \"What Python PEPs were accepted in 2026?\", \"tools\": [{\"type\": \"web_search\", \"search_context_size\": \"high\", \"filters\": {\"allowed_domains\": [\"python.org\", \"peps.python.org\"]}, \"user_location\": {\"type\": \"approximate\", \"country\": \"US\"}}]}" \
    expect_completed.py

# --- Negative tests ---

run_test "Test 6: Invalid search_context_size rejected" \
    "{\"model\": \"$MODEL\", \"input\": \"hello\", \"tools\": [{\"type\": \"web_search\", \"search_context_size\": \"ultra\"}]}" \
    expect_validation_error.py

run_test "Test 7: Invalid tool type rejected" \
    "{\"model\": \"$MODEL\", \"input\": \"hello\", \"tools\": [{\"type\": \"web_search_nonexistent\"}]}" \
    expect_validation_error.py

run_test "Test 8: Empty allowed_domains still works" \
    "{\"model\": \"$MODEL\", \"input\": \"What is the current price of Bitcoin today?\", \"tools\": [{\"type\": \"web_search\", \"filters\": {\"allowed_domains\": []}}]}" \
    expect_completed.py

# --- Behavioral verification tests ---

# Verify search_context_size actually controls source count (low=3 vs high=10)
cat > "$CHECKER_DIR/compare_source_counts.py" << 'PYEOF'
import sys, json

low_data = json.load(sys.stdin)
high_data = json.loads(sys.argv[1])

def get_source_count(data):
    total = 0
    for o in data.get("output", []):
        if o.get("type") == "web_search_call":
            action = o.get("action")
            if action:
                total += len(action.get("sources", []))
    return total

low_count = get_source_count(low_data)
high_count = get_source_count(high_data)
print(f"  low sources:  {low_count}")
print(f"  high sources: {high_count}")
if low_count == 0 and high_count == 0:
    print("  => FAIL (no sources in either response)")
    sys.exit(1)
if high_count >= low_count:
    print("  => PASS (high >= low)")
    sys.exit(0)
else:
    print("  => FAIL (high < low, search_context_size not effective)")
    sys.exit(1)
PYEOF

echo "=== Test 9: search_context_size controls source count (low vs high) ==="
QUERY="What are the latest developments in quantum computing?"
LOW_RESP=$(curl -s "$BASE_URL/v1/responses" \
    -X POST -H "Content-Type: application/json" \
    -d "{\"model\": \"$MODEL\", \"input\": \"$QUERY\", \"tools\": [{\"type\": \"web_search\", \"search_context_size\": \"low\"}]}")
HIGH_RESP=$(curl -s "$BASE_URL/v1/responses" \
    -X POST -H "Content-Type: application/json" \
    -d "{\"model\": \"$MODEL\", \"input\": \"$QUERY\", \"tools\": [{\"type\": \"web_search\", \"search_context_size\": \"high\"}]}")
echo "$LOW_RESP" | python3 "$CHECKER_DIR/compare_source_counts.py" "$HIGH_RESP" \
    && pass=$((pass + 1)) || fail=$((fail + 1))
echo ""

# Verify allowed_domains actually restricts sources
cat > "$CHECKER_DIR/verify_domain_filter.py" << 'PYEOF'
import sys, json

data = json.load(sys.stdin)
allowed = json.loads(sys.argv[1])

urls = []
for o in data.get("output", []):
    if o.get("type") == "web_search_call":
        action = o.get("action")
        if action:
            for s in action.get("sources", []):
                urls.append(s.get("url", ""))

if not urls:
    print("  no sources returned")
    print("  => FAIL")
    sys.exit(1)

violations = []
for url in urls:
    if not any(domain in url for domain in allowed):
        violations.append(url)

print(f"  sources: {len(urls)}")
for u in urls[:5]:
    match = any(d in u for d in allowed)
    print(f"    {'OK' if match else 'BAD'} {u}")
if violations:
    print(f"  {len(violations)} source(s) outside allowed domains")
    print("  => FAIL")
    sys.exit(1)
else:
    print("  all sources match allowed domains")
    print("  => PASS")
    sys.exit(0)
PYEOF

echo "=== Test 10: allowed_domains restricts returned sources ==="
curl -s "$BASE_URL/v1/responses" \
    -X POST -H "Content-Type: application/json" \
    -d "{\"model\": \"$MODEL\", \"input\": \"What is new in Python 3.13?\", \"tools\": [{\"type\": \"web_search\", \"search_context_size\": \"high\", \"filters\": {\"allowed_domains\": [\"python.org\", \"docs.python.org\"]}}]}" \
    | python3 "$CHECKER_DIR/verify_domain_filter.py" '["python.org", "docs.python.org"]' \
    && pass=$((pass + 1)) || fail=$((fail + 1))
echo ""

echo "================================"
echo "Results: $pass passed, $fail failed"
exit $((fail > 0 ? 1 : 0))

[github-actions]

✱ Stainless preview builds

This PR will update the llama-stack-client SDKs with the following commit message.

fix(brave-search): align result slicing with search_context_size count

⚠️ llama-stack-client-openapi studio · code

Your SDK build had at least one "warning" diagnostic.
generate ⚠️

⚠️ llama-stack-client-python studio · code

Your SDK build had at least one "warning" diagnostic.
generate ⚠️ → build ✅ → lint ✅ → test ✅

pip install https://pkg.stainless.com/s/llama-stack-client-python/40f71628597b92ed14c6ae3445e09ffb2927da53/ogx_client-0.8.0a2-py3-none-any.whl

⚠️ llama-stack-client-go studio · conflict

Your SDK build had at least one warning diagnostic.

⚠️ llama-stack-client-node studio · code

Your SDK build had at least one "warning" diagnostic.
generate ⚠️ → build ✅ → lint ✅ → test ✅

npm install https://pkg.stainless.com/s/llama-stack-client-node/0b0984ead8282130b2f6f5380736753e957a4d8b/dist.tar.gz

---

This comment is auto-generated by GitHub Actions and is automatically kept up to date as you push.
If you push custom code to the preview branch, re-run this workflow to update the comment.
Last updated: 2026-05-27 15:36:40 UTC

[leseb]

🤟🏻 🔥

[mergify]

This pull request has merge conflicts that must be resolved before it can be merged. @leseb please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 5 additional findings.

[franciscojavierarceo]

thanks devin