GPT-4.1 JSON response produces null bytes instead of valid unicode strings for large inputs

[wnolden]

Hi there,

I am using the openai-go SDK to parse some inputs with the GPT-4.1 model that contain special characters like ä, ü, ñ etc. with a fixed response schema like this:

var schema = map[string]any{
	"type": "object",
	"properties": map[string]any{
		"certificates": map[string]any{
			"type": "array",
			"items": map[string]any{
				"type": "object",
				"properties": map[string]any{
					"name":    map[string]any{"type": "string"},
					"context": map[string]any{"type": "string"},
				},
				"required":             []string{"name", "context"},
				"additionalProperties": false,
			},
		},
	},
	"required":             []string{"certificates"},
	"additionalProperties": false,
}

// ...

text := "Some Text with special characters. Certificates include: Datenschutzbeauftragter TÜV SÜD (2021)"

resp, err := client.Chat.Completions.New(ctx, openai.ChatCompletionNewParams{
	Model:       modelName,
	Temperature: param.NewOpt(0.0),
	Messages: []openai.ChatCompletionMessageParamUnion{
		openai.SystemMessage("Extract certificates from the following text."),
		openai.UserMessage(fmt.Sprintf("**Text:**\n\"\"\"%s\"\"\"", text)),
	},
	ResponseFormat: openai.ChatCompletionNewParamsResponseFormatUnion{
		OfJSONSchema: &shared.ResponseFormatJSONSchemaParam{
			JSONSchema: shared.ResponseFormatJSONSchemaJSONSchemaParam{
				Name:   "CertificatesSchema",
				Strict: param.NewOpt(true),
				Schema: schema,
			},
		},
	},
})

This produces a valid Unicode response:

content := resp.Choices[0].Message.Content
fmt.Printf("%q\n\n", content)
// Output: "{\"certificates\":[{\"context\":\"Certificates include: Datenschutzbeauftragter TÜV SÜD (2021)\",\"name\":\"Datenschutzbeauftragter TÜV SÜD\"}]}"

However, on very very long texts with the GPT-4.1 model specifically this does not work and produces the following:

text := "Some Text with special characters. Certificates include: Datenschutzbeauftragter TÜV SÜD (2021) ... (Add about ~10KB worth of characters here)"

// ...

content := resp.Choices[0].Message.Content
fmt.Printf("%q\n\n", content)
// Output: "{\"certificates\":[{\"context\":\"Certificates include: Datenschutzbeauftragter T\\x00dcV S\\x00dcD (2021)\",\"name\":\"Datenschutzbeauftragter T\\x00dcV S\\x00dcD\"}]}"

Instead of outputting a \u00dc, the model generates a \u0000dc, where the \u0000 is interpreted as a null byte. When continuing with this the rest of the string becomes corrupted.

I got the following function working to sanitize the output beforehand, but it seems very expensive to do frequently on large responses:

// No clue if that is the best approach

var nullEscapeRe = regexp.MustCompile(`\\u0000([0-9a-fA-F]{2})`)

func sanitizeNullEscapes(raw string) string {
	if !strings.Contains(raw, `\u0000`) {
		return raw
	}
	fixed := nullEscapeRe.ReplaceAllString(raw, `\u00$1`)
	return fixed
}

content = sanitizeNullEscapes(content)

This does not happen with the GPT-5.4 model, however I am bound by external guidelines and I have to use 4.1, at least for now. Is there a workaround that does not involve me manually fixing the output?

I apologize in advance if this SDK is the wrong place to open this issue, if this is an underlying issue with the model itself feel free to close this issue 😊

[rsd-darshan]

I dug into this. The relevant decode path is internal/apijson/decoder.go, which uses tidwall/gjson to pull string values out of the response JSON.

I reproduced the case from the issue with a minimal example: JSON containing a NUL escape immediately followed by the characters d and c, which is what you'd see if the model meant to emit Ü but instead emitted a NUL and let dc leak out as literal text.

Result: gjson.String() and Go's standard library encoding/json.Unmarshal decode this byte-for-byte identically — both produce a string containing a literal NUL byte followed by d and c. There's no divergence between them, and no corruption introduced by the SDK's decoder.

A NUL escape is a syntactically valid JSON escape for the NUL character, so any compliant JSON parser — gjson, encoding/json, or otherwise — has to decode it exactly this way. The SDK has no way to distinguish "the model meant a NUL byte" from "the model meant a different character and emitted a malformed escape by mistake" — both look like syntactically valid JSON from the parser's side.

Given that GPT-5 doesn't reproduce this and the behavior looks specific to GPT-4.1's generation, this looks like a model output quality issue rather than something fixable in the SDK's JSON decoding layer. I don't think there's a safe code change to make here without guessing at intent, which would risk silently rewriting bytes for other valid use cases.