callback: added a pre-hook callback to remove anthropic header#98
callback: added a pre-hook callback to remove anthropic header#98SirajuddinShaik wants to merge 6 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a production pre-call guardrail callback intended to strip Anthropic billing-header metadata from request payloads to improve cache hit rates and avoid cache-busting content in prompts.
Changes:
- Introduces
MessageFilterProd(CustomGuardrail) with anasync_pre_call_hookto filter matching content fromdata["messages"]anddata["system"]. - Adds a pre-configured production instance
message_filter_prod.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
| def _filter_content(self, content): | ||
| """Filter content (string or list of blocks).""" | ||
| if isinstance(content, str): | ||
| return content | ||
|
|
There was a problem hiding this comment.
_filter_content() returns strings unchanged, so this hook will never remove the x-anthropic-billing-header: metadata when it appears as a string (e.g., system message content can be a plain string in existing Anthropic translation paths). To actually prevent cache-busting, add string handling that detects/removes (or returns None for) strings starting with the billing-header prefix and ensure callers drop the field/message when the result is None.
There was a problem hiding this comment.
@copilot open a new pull request to apply changes based on this feedback
| filtered_content = self._filter_content(content) | ||
| if filtered_content != content: | ||
| message["content"] = filtered_content | ||
| filtered_count += 1 |
There was a problem hiding this comment.
If all content blocks are filtered out, filtered_content becomes an empty list and the message is still kept with content=[]. Several provider adapters (including Anthropic) treat empty text blocks/content as invalid and can error. After filtering, if content becomes empty (or None if you adopt that), remove the entire message entry (and similarly clear data['system']) instead of leaving an empty content value.
| async def async_pre_call_hook( | ||
| self, | ||
| user_api_key_dict: UserAPIKeyAuth, | ||
| cache: DualCache, | ||
| data: dict, | ||
| call_type: CallTypesLiteral, | ||
| ) -> Optional[Union[Exception, str, dict]]: | ||
| """Filter cache-busting content from messages and system fields.""" |
There was a problem hiding this comment.
This PR adds new behavior that mutates request payloads in a pre-call hook but doesn’t include tests. Please add at least one unit test covering: (1) filtering when the header appears in string content (system message/content), (2) filtering when it appears as a [{type:'text', text:'...'}] block list, and (3) the empty-after-filter case to ensure the message/system field is removed rather than left empty.
| # Production instance | ||
| message_filter_prod = MessageFilterProd(filter_keywords=["x-anthropic-billing-header"]) |
There was a problem hiding this comment.
The default/production filter_keywords uses the prefix "x-anthropic-billing-header" without the trailing :. This will also match and remove any user content that happens to begin with that phrase (not necessarily the metadata line). Consider using the more specific prefix used elsewhere in the codebase ("x-anthropic-billing-header:") or otherwise tightening the match to reduce false positives.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
|
@SirajuddinShaik I've opened a new pull request, #99, to work on those changes. Once the pull request is ready, I'll request review from you. |
|
Duplicate functionality: This filtering already exists in two places:
|
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
Relevant issues
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unitCI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes