Control-Layer Throwing Unterminated JSON Errors in Playground and for API Clients (Back-End does not)

[sempervictus]

Seeing this on every model as of last night/today's docker image:

Other clients have no issues streaming with the back-ends to which control-layer is speaking.

[linear]

from peter:

Hey, we'll look into this and report back thanks for raising!

[sempervictus]

Thank you.

Quick update on this: i can verify its happening in DWCL by running the same query to the same phi through the control-layer and then through the tensorzero behind it:

$ aichat -m dw:phi4 evaluate and explain your capabilities
As a language model AI, developed by Microsoft, I have several capabilities centered around processing and generating human language text. Here’s an overview of what I can do:

1. **Information Retrieval and Explanation
   Error: Failed to call chat-completions api

Caused by:
EOF while parsing a string at line 1 column 31

vs hitting the API endpoint at which control-layer is pointing to talk to the models directly:

$ aichat -m tz:tensorzero::model_name::phi4 evaluate and explain your capabilities
As a language model developed by OpenAI, I'm designed to assist users by providing information, answering questions, and offering explanations across a wide range of topics. Here are some of my core capabilities and limitations:

Capabilities:

1. Information Retrieval and Explanation:
   • I can provide information on a vast array of subjects including science, history, technology, and more.
   • I can explain complex concepts in simpler terms to aid understanding.

.

6. Accuracy and Verification:
   • While I strive for accuracy, I cannot verify the information I provide. Users should cross-check important or sensitive information with reliable sources.

Overall, while I can be a helpful tool for a wide range of tasks, it's important to use me as a complement to, rather than a replacement for, human expertise and critical thinking.

bypassing TZ direct to the vllm.rs endpoint behind it succeeds too whereas control-layer fails both to communicate with clients passing through it and its own playground

[sempervictus]

Works when streaming is disabled by the way - so this is an SSE issue

[pjb157]

Hey @sempervictus thanks for the additional information. We'll get to this either backend of this week or next once we've cleared our current load. I'll report back progress but appreciate your help!

[sempervictus]

Im guessing this is related to the redaction change in the underlying streaming lib

[sempervictus]

Interestingly this might be a phi4-specific concern, i've a bunch of Qwen3s running which do not exhibit this behavior but its definitely in the processing stack of DW since TZ and the the actual inferencing API dont produce malformed streams

[sejori]

Hi there @sempervictus - I have fixed our onwards crate to support a buffer of incomplete SSE data for each request. Control layer is now running this latest version so the issue should be resolved for you. Please let us know if it persists.

[sempervictus]

@sejori - updated to the latest container and oddly enough... now the problem seems to manifest with a Qwen3 on the back-end. Testing gemma3 and phi4 currently

Something went wrong. Here's the specific error message we encountered: An error occurred while processing the request: Unterminated string in JSON at position 40

from clients in front of it and its own playground