feat: add /tokenize endpoint for openai-compatible backend

[hustxiayang]

Description
This is split from #1650. Specifically this is to support tokenization for openai-compatible backend.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 87.01923% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.84%. Comparing base (7113614) to head (bbd7182).

Files with missing lines	Patch %	Lines
internal/translator/tokenize.go	85.18%	4 Missing and 4 partials ⚠️
internal/apischema/openai/tokenize/tokenize.go	82.85%	3 Missing and 3 partials ⚠️
internal/endpointspec/endpointspec.go	81.81%	4 Missing ⚠️
internal/tracing/openinference/openai/tokenize.go	94.80%	2 Missing and 2 partials ⚠️
internal/tracing/tracer.go	66.66%	2 Missing and 1 partial ⚠️
internal/tracing/tracing.go	77.77%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2371      +/-   ##
==========================================
+ Coverage   84.82%   84.84%   +0.02%     
==========================================
  Files         148      151       +3     
  Lines       21878    22086     +208     
==========================================
+ Hits        18558    18739     +181     
- Misses       2198     2215      +17     
- Partials     1122     1132      +10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[hustxiayang]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds support for a new /tokenize endpoint compatible with the vLLM tokenize API, including request/response schemas, endpoint parsing, translation, metrics, tracing, and documentation. The reviewer feedback highlights a critical issue where boolean fields that default to true (such as AddSpecialTokens and AddGenerationPrompt) are defined as plain bool with omitzero tags, which would cause explicit false values to be omitted during marshaling. The reviewer recommends changing these fields to *bool pointers and provides the necessary updates for validation, tracing, and unit tests to safely handle the pointer dereferencing.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[hustxiayang]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for a new /tokenize endpoint compatible with the vLLM tokenize API, allowing users to count tokens for both chat and completion requests. It adds the necessary API schemas, endpoint parsing, metrics, OpenInference tracing, and translation logic, along with comprehensive tests and documentation. The review feedback highlights several key improvement opportunities, including strengthening request validation to handle default values for conflicting fields, adding nil checks to prevent potential panics, avoiding implicit struct field promotion in tracing attributes, and ensuring robust error translation when the Content-Type header is missing.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[hustxiayang]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds support for a new /tokenize endpoint compatible with the vLLM tokenize API, including request/response schemas, endpoint specifications, OpenInference tracing, and translation logic. The review feedback highlights several improvement opportunities: optimizing JSON parsing in ParseBody by using a lightweight detector struct to avoid double unmarshaling, ensuring RequestUnion.Validate() triggers underlying ChatRequest validation, consistently accessing ChatRequest fields explicitly to prevent potential panics, and populating tokenUsage in ResponseBody to correctly track token usage metrics as documented.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[hustxiayang]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for a new /tokenize endpoint compatible with the vLLM tokenize API, allowing token counting for both chat messages and completion prompts. The changes span endpoint registration, API schema definitions, translation logic, metrics, tracing, and documentation. The review feedback highlights several opportunities to improve robustness by adding defensive nil checks for request, response, and body readers, as well as explicitly accessing embedded struct fields to avoid ambiguity.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[hustxiayang]

/retest

[aabchoo]

Overall looks good to me!