GLM-OCR: /chat/completions produces repetition loops on prose; repetition_penalty does not prevent them

[esaruoho]

Summary

When OCR'ing a scanned page of flowing English prose through the mlx-vlm server (/chat/completions) with mlx-community/GLM-OCR-bf16, the model locks into a repetition loop after the first ~500 correct characters and re-emits the same short paragraph dozens of times until max_tokens truncates. Setting repetition_penalty on the request does not prevent this.

The same source image processed by the same model weights via Ollama (GGUF quantisation, llama.cpp sampler) produces clean, coherent OCR with no loops — so the model itself and image understanding are fine; the issue is isolated to mlx-vlm's generation / sampling path.

Environment

• mlx-vlm 0.4.4 (installed from git+https://github.com/Blaizzy/mlx-vlm.git)
• Python 3.12.13 (Homebrew, macOS arm64)
• macOS Darwin 24.6.0, Apple Silicon Mac Mini
• torch 2.11.0 + torchvision 0.26.0 installed (required for GlmOcrProcessor via closed issue GLM-OCR: images not processed (GlmOcrProcessor not loaded by AutoProcessor) #886)
• Model: mlx-community/GLM-OCR-bf16 (also reproduces with EZCon/GLM-OCR-mlx)

Reproduction

1. Start the server:

   python -m mlx_vlm.server --trust-remote-code --port 8080
   

2. Render a page of flowing English prose at 200 DPI to PNG (image-only scanned PDF, typical size ~300–500 KB). A public-domain Project Gutenberg scan reproduces the issue on any page with an all-caps section heading.

3. POST to /chat/completions:

   {
     "model": "mlx-community/GLM-OCR-bf16",
     "messages": [{"role": "user", "content": [
       {"type": "image_url", "image_url": {"url": "data:image/png;base64,<...>"}},
       {"type": "text", "text": "OCR this image. Extract ALL text preserving the original formatting, paragraphs, tables, and formulas. Output only the extracted text."}
     ]}],
     "max_tokens": 4096,
     "temperature": 0,
     "repetition_penalty": 1.1
   }

4. Observe: the first ~500 characters of the response match the page content correctly. Then the model hits a short all-caps heading (e.g. a section header, ~3 words) and begins re-emitting the heading + the subsequent paragraph in a loop until max_tokens.

Measured behaviour

Same 200-DPI rendering, single page of English prose, identical prompt:

repetition_penalty	temperature	top_p	Output (chars)	Repeated-heading occurrences
unset	0.0	—	33,688	~100
1.05	0.0	—	16,870	54
1.1	0.0	—	16,870	54
1.3	0.3	—	16,657	79
1.5	0.0	—	16,594	80
1.1	0.5	0.9	17,086	54

Ground truth (Ollama GGUF on same image, glm-ocr:latest, default sampler): 1,603 chars, heading appears once. Complete correct transcription of the page.

Observations:

• repetition_penalty values from 1.05 to 1.5 do not break the loop on this content
• Increasing repetition_penalty from 1.1 → 1.5 at temperature=0 actually increased repeat count (54 → 80)
• Temperature > 0 + top_p 0.9 did not help either
• The loop output consistently fills close to max_tokens (input ~5000 prompt tokens, output ~4096 generation tokens)

Hypothesis

Either:

1. The repetition_penalty field on /chat/completions is accepted by the request model but not threaded into stream_generate / generate_step for VLM paths, or
2. It is applied but using a codebase convention (e.g. a context window length) that renders it ineffective for this model's tokenisation of repeated all-caps headings, or
3. GLM-OCR's chat template's pre-filled <think></think> interacts with sampling in a way that bypasses the penalty.

The counterfactual (Ollama/llama.cpp producing clean output from the same weights on the same image) strongly points at the mlx-vlm sampling path as the isolated cause.

Related

• GLM-OCR: images not processed (GlmOcrProcessor not loaded by AutoProcessor) #886 (closed) — GlmOcrProcessor not loaded by AutoProcessor; same model, different failure mode. Resolved by installing torch+torchvision, but exposed this downstream sampler issue.
• CRITICAL ISSUE: GENERATION IS WRONG AND DIFFERENT FROM LLAMA CPP #387 (closed, different model) — similar pattern of mlx-vlm generation diverging from llama.cpp.

Why this matters

GLM-OCR via mlx-vlm is otherwise an excellent fit for Apple Silicon OCR pipelines (~2× throughput vs. Ollama on benchmarks). This sampler issue makes it unusable today for anyone processing long-form prose; structured content (receipts, forms, equations) appears to mask the problem because short repetitive headers are rarer there.

Happy to test patches against the same reproduction and report back. I'm also skimming the sampling code to see whether a small PR is feasible; will link if I find something.

Thanks for maintaining this project!

[esaruoho]

Update: deeper investigation results

I instrumented mlx_vlm.generate.generate_step and mlx_lm.sample_utils.make_repetition_penalty to trace what actually happens, and rebenchmarked across a wider parameter grid. Posting results in case they're useful.

Plumbing is correct

The repetition_penalty and repetition_context_size parameters are both correctly threaded from the request → build_generation_kwargs → generate() → stream_generate() → generate_step() → make_logits_processors() → the penalty closure. Debug log:

[DBG] make_logits_processors called: penalty=1.1, context_size=4096
[DBG-make_rep_pen] FACTORY called with penalty=1.1, context_size=4096
[DBG-rep_pen] CALLED tokens.shape=(1,), will use last context_size=4096
[DBG-rep_pen] CALLED tokens.shape=(2,), will use last context_size=4096
...

So repetition_context_size is not being silently dropped (my initial hypothesis was wrong on that point).

Token accumulation observation

tokens grows from (1,) upward, by one per generated token — the prompt tokens are not included in the penalty's scan history. This is a side-effect of the chunked prefill at generate.py:549:

input_ids = input_ids[:, -1:]   # ← trims prompt to just the last token
y, logprobs = _step(input_ids, inputs_embeds=inputs_embeds)

For long prompts (image embeddings, ~5000 tokens here) the entire prompt is dropped from the sampler's view of the token history. Probably intentional (we don't want to penalize tokens that appear in the prompt OCR-ed page text), but worth knowing.

Why repetition_context_size didn't matter in my earlier table

At temperature=0 with repetition_penalty ≤ 1.5, the loop tokens' logits are dominant enough that even maximally widening the context window doesn't change the argmax. Output was byte-identical (16,870 chars, 54 MIRACLE) for repetition_context_size ∈ {20, 512, 2048, 4096}. Widening the window adds more unique tokens to the penalty set, but each unique token is penalized exactly once regardless of frequency, so the dominant loop tokens still win.

What actually breaks the loop

repetition_penalty=2.0, temperature=0:    660 chars,   0 MIRACLE  ✅
repetition_penalty=5.0, temperature=0:   1085 chars,   1 MIRACLE  ✅
repetition_penalty=10.0, temperature=0:  1085 chars,   1 MIRACLE  ✅

repetition_penalty=2.0 fully eliminates the loop on this content. Ollama's clean baseline on the same image is 1,603 chars; MLX at rp=2.0 produces a slightly truncated but otherwise correct transcription.

Suggested refinements (potentially worth a PR)

1. Expose frequency_penalty and presence_penalty in GenerationParams and forward them through generate_step to make_logits_processors. They're already supported by the underlying mlx_lm.sample_utils.make_logits_processors, just not surfaced. Frequency penalty (additive, scales with count) is the natural fix for "loops dominated by a few tokens" — gives users the right tool without needing a huge multiplicative penalty.

2. Expose repetition_context_size so callers can opt into a wider lookback (helpful when using frequency_penalty/presence_penalty alongside).

3. Document the rp ≥ 2.0 caveat for VLM/OCR: the default rp=1.1 inherited from text-LLM convention is below the threshold needed for VLMs whose logits are sharply peaked.

Happy to open a PR doing (1) and (2). Will leave (3) to the maintainers' preference.

Workaround for users today

Set repetition_penalty: 2.0 in your /chat/completions request body. It works.