Skip to content

fix: remove Anthropic structured outputs beta header (now GA)#4987

Open
Wh1isper wants to merge 3 commits intopydantic:mainfrom
Wh1isper:remove-structured-outputs-beta-header
Open

fix: remove Anthropic structured outputs beta header (now GA)#4987
Wh1isper wants to merge 3 commits intopydantic:mainfrom
Wh1isper:remove-structured-outputs-beta-header

Conversation

@Wh1isper
Copy link
Copy Markdown
Contributor

@Wh1isper Wh1isper commented Apr 5, 2026

Anthropic structured outputs (native JSON schema output and strict tool calls) have graduated from beta to GA. The beta header is no longer required and the output_config parameter is already used (the GA name).

See: https://platform.claude.com/docs/en/build-with-claude/structured-outputs

Changes:

  • Remove automatic addition of structured-outputs-2025-11-13 beta header in _get_betas_and_extra_headers()
  • Simplify method signature by removing unused tools and model_request_parameters parameters
  • Remove beta header verification hook from test_output.py (as noted by the existing TODO comment: "remove when structured outputs is generally available")
  • Update test assertions and docstrings accordingly

Pre-Review Checklist

  • Any AI generated code has been reviewed line-by-line by the human PR author, who stands by it.
  • No breaking changes in accordance with the version policy.
  • Linting and type checking pass per make format and make typecheck.
  • PR title is fit for the release changelog.

Pre-Merge Checklist

  • New tests for any fix or new behavior, maintaining 100% coverage.
  • Updated documentation for new features and behaviors, including docstrings for API docs.

@github-actions github-actions bot added size: M Medium PR (101-500 weighted lines) chore labels Apr 5, 2026
@Wh1isper Wh1isper force-pushed the remove-structured-outputs-beta-header branch from 33ff459 to 0f08b27 Compare April 5, 2026 15:59
Co-Authored-By: YAAI <261597362+yaai-bot@users.noreply.github.com>
@Wh1isper Wh1isper force-pushed the remove-structured-outputs-beta-header branch from 0f08b27 to fb33281 Compare April 5, 2026 16:05
@Wh1isper Wh1isper changed the title Remove structured-outputs-2025-11-13 beta header for Anthropic Remove Anthropic structured outputs beta header (now GA) Apr 5, 2026
@Wh1isper Wh1isper changed the title Remove Anthropic structured outputs beta header (now GA) fix: remove Anthropic structured outputs beta header (now GA) Apr 5, 2026
@Wh1isper
Copy link
Copy Markdown
Contributor Author

Wh1isper commented Apr 5, 2026

 {"type":"error","error":{"type":"invalid_request_error","message":"Unexpected value(s) `structured-outputs-2025-11-13` for the `anthropic-beta` header. Please consult our documentation at docs.anthropic.com or try again without the header."},"request_id":"req_vrtx_011CZks2kZvkrRPWnpxrNBtH"}

Using this header works fine with the official Anthropic API, but I encountered this error when using an internal gateway to route requests from Anthropic to the Anthropic model on Vertex AI. This may be due to inconsistencies in how the header behaves between the official Anthropic API and Vertex AI.

devin-ai-integration[bot]

This comment was marked as resolved.

@Kludex Kludex self-requested a review April 6, 2026 09:35
@Kludex
Copy link
Copy Markdown
Member

Kludex commented Apr 6, 2026

@Wh1isper Can we have a VCR update?

@Wh1isper
Copy link
Copy Markdown
Contributor Author

Wh1isper commented Apr 6, 2026

I ran the full test suite against the existing cassettes and all 24 tests in test_output.py pass without changes. The cassettes don't need updating because the anthropic-beta header is already stripped by the serializer (FILTERED_HEADER_PREFIXES = ['anthropic-', ...] in json_body_serializer.py), and the ?beta=true query param comes from the client.beta.messages SDK namespace rather than the betas list, so cassette URIs are unaffected.

I don't have an official Anthropic API key, but I can record via a gateway if needed. Let me know if you'd like me to add additional VCR coverage for the beta header scenarios specifically.

@Kludex
Copy link
Copy Markdown
Member

Kludex commented Apr 6, 2026

In this case, I find it useful to record the cassette again. We want to confirm the behavior of the provider itself. The cassettes don't change because they don't care about headers matching.

@adtyavrdhn Can you find a useful test that was using this functionality and record the cassette again, please? One is fine.

Copy link
Copy Markdown
Member

@Kludex Kludex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good clean up here. 👍

Approving pending the cassette update.

Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

View 6 additional findings in Devin Review.

Open in Devin Review

Comment on lines +214 to 217
"""Agent with no tools and no output_type."""
model = anthropic_model('claude-sonnet-4-5')
hook = create_header_verification_hook(expect_beta=False, test_name='test_no_tools_no_output')
model.client._client.event_hooks['request'].append(hook) # pyright: ignore[reportPrivateUsage]

agent = Agent(model)
agent.run_sync('Tell me a brief fact about Paris')
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 Many VCR integration tests now have zero explicit assertions

After removing the create_header_verification_hook checks, 11 tests in tests/models/anthropic/test_output.py now have no assertions at all — they just call agent.run_sync() and return. For example, test_no_tools_no_output (line 214-217), test_strict_true_tool_no_output (line 278-286), test_mixed_tools_no_output (line 412-424), etc. The VCR cassettes only match on HTTP method and URL path (see custom matchers at tests/conftest.py:323-335), so they don't validate request body content. These tests now effectively only verify that the code doesn't crash. The tests/AGENTS.md rule:334 says to "assert meaningful behavior in tests, not just code execution." While removing the beta header assertions is correct (the feature is GA), these tests could benefit from either (a) being removed entirely if they no longer test distinct behavior, or (b) having new assertions added (e.g., verifying tool schemas, output_config presence, or snapshotting result.all_messages() per rule:194).

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore size: M Medium PR (101-500 weighted lines)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remove structured-outputs-2025-11-13 beta header (now GA)

3 participants