fix(responses): delimit untrusted web_search/file_search tool output before feeding it back to the model

[mittalpk]

Fixes #6263

What

web_search, file_search, and knowledge_search results were placed into the model's next-turn context verbatim, with no boundary between trusted instructions and untrusted, externally-sourced content (scraped web pages, indexed documents). An attacker who controls a page that gets searched or indexed could inject text the model treats as an instruction rather than as data (indirect prompt injection).

Fix

Wraps the text portions of results from these three tools in explicit <untrusted_tool_output> delimiters with a short instruction that the enclosed content is untrusted data to analyze, never instructions to follow. MCP tool output and other tool types are unaffected, matching the scope of the reported issue. Image content parts pass through unwrapped.

Two additional issues surfaced during an edge-case pass over this same code path and are fixed here too, since they live in the exact function this PR already touches:

1. Delimiter-collision escaping. Content containing a literal </untrusted_tool_output> could close the delimited block early and make injected text that follows look like it sits outside the untrusted region — defeating the wrapping with itself. Both tags are now escaped inside untrusted content before wrapping, case-insensitively (case variation like </UNTRUSTED_TOOL_OUTPUT> is a trivial, well-known evasion of a naive case-sensitive match).
2. Empty results reported as failure. A successful search that legitimately returns empty content (zero results) was fed to the model as "Tool execution failed", because the pre-existing check used truthiness (if result and result_content:) rather than distinguishing "no result at all" from "a result with empty content." Changed to an explicit is not None check.

Known limitation (documented, not a blocker)

Whitespace-padded tag variants (e.g. < /untrusted_tool_output >) and Unicode-homoglyph tricks are not caught by the current escaping — closing that fully would need a structurally different defense (e.g. a per-request random delimiter token instead of a static string). Noted explicitly in the _escape_delimiter_collisions docstring as a reasonable follow-up rather than silently left unstated.

Tests

New tests/unit/providers/inline/responses/builtin/responses/test_tool_executor.py (13 tests, this module previously had zero coverage):

• Delimiting applied correctly for web_search/file_search/knowledge_search, both string and mixed text+image list content shapes.
• MCP tool output confirmed not wrapped (out of scope).
• Delimiter-collision escaping, including the case-insensitivity fix, confirmed to neutralize an embedded fake close-tag without breaking the real one.
• Empty-content-not-reported-as-failure, with a sanity check that a genuinely missing result (result=None) still correctly reports failure.

One existing test (test_openai_responses_tools.py::test_create_openai_response_with_string_input_with_tools) asserted tool output survives verbatim (content == "Dublin"); updated to "Dublin" in content since that's the new, correct contract given the delimiting.

Every new/changed assertion was confirmed to fail against the pre-fix code (via git stash) before the corresponding fix landed.

Verification

• uv run pytest tests/unit/providers/inline/responses/builtin/ tests/unit/providers/responses/builtin/: 373 passed, no regressions
• uv run ruff check / ruff format --check: clean
• uv run mypy: no issues
• uv run pre-commit run --files <changed files>: all hooks passed (license header, FIPS check, SQL-injection lint, logging conventions, codegen-drift checks all N/A since no API/provider schema was touched)

[mattf]

thanks for helping us make progress in this game of cat and mouse.

nit: use narrower types than Any

[mattf]

@nathan-weinberg ptal

[mittalpk]

@mattf, Thank you

The two failing checks (azure/responses, gpt/responses) are expected — this PR changes the exact request content sent to the model for web_search/file_search tool results (that's the fix), which invalidates the recorded fixtures keyed by request hash for those two suites. Since this is a fork PR, the auto-recording workflow only ran ollama (no secrets) and skipped gpt/azure.

Could you trigger Integration Tests (Record) via workflow_dispatch with providers: gpt,azure and pr_number: 6337? That should refresh the affected recordings and auto-commit them back to this branch via commit-recordings.yml.

[github-actions]

Recording workflow completed

Providers: gpt, azure

Recordings have been generated and will be committed automatically by the companion workflow.

View workflow run

Fork PR: Recordings will be committed if you have "Allow edits from maintainers" enabled.

[github-actions]

✅ Recordings committed successfully

Recordings from the integration tests have been committed to this PR.

View commit workflow