bug: gemma4 infinitely repeating a token randomly...

[onzag]

Issue description

gemma4 infinitely repeating a token randomly...

Expected Behavior

Not repeating the same token or phrase forever

Actual Behavior

This might be related.

ggml-org/llama.cpp#21726

Thinking is off.

No grammar, in fact the one with grammar seems to be fine.

Tried on many gemma 4 versions, with the same issue.

Steps to reproduce

Tried this one
https://huggingface.co/unsloth/gemma-4-31B-it-GGUF/blob/main/gemma-4-31B-it-UD-Q8_K_XL.gguf

and various of
https://huggingface.co/mradermacher/Gemma-4-31B-StyleTune-GGUF

The bug seems to happen, randomly, so there's no reliable way to generate it.

My Environment

Dependency	Version
Operating System	Windows
CPU	ryzen 5
Node.js version	24.11.1
node-llama-cpp version	3.19.0

npx --yes node-llama-cpp inspect gpu output:

Result of running `npx --yes node-llama-cpp inspect gpu`

OS: Windows 10.0.26200 (x64)
Node: 24.11.1 (x64)

node-llama-cpp: 3.19.0
Prebuilt binaries: b9842

CUDA: available
Vulkan: available

CUDA device: NVIDIA GeForce RTX 3090
CUDA used VRAM: 5.19% (1.25GB/24GB)
CUDA free VRAM: 94.8% (22.75GB/24GB)

Vulkan devices: NVIDIA GeForce RTX 3090, AMD Radeon(TM) Graphics
Vulkan used VRAM: 4.66% (2.64GB/56.58GB)
Vulkan free VRAM: 95.33% (53.94GB/56.58GB)
Vulkan unified memory: 21.97GB (38.82%)

CPU model: AMD Ryzen 7 9800X3D 8-Core Processor
Math cores: 8
Used RAM: 27.35% (16.86GB/61.66GB)
Free RAM: 72.64% (44.8GB/61.66GB)
Wired RAM: 2.07% (1.28GB/61.66GB)
Used swap: 46.96% (33.28GB/70.86GB)
Max swap size: 70.86GB
mmap: supported

Additional Context

I think this is about an underlying llama.cpp bug or something akin.

Relevant Features Used

• Metal support
• CUDA support
• Vulkan support
• Grammar
• Function calling

Are you willing to resolve this issue by submitting a Pull Request?

Yes, I have the time, and I know how to start.

[onzag]

I just cannot find a proper way to replicate this, some grammars make the issue more likely, others not, sometimes lack of grammar makes it more likely, longer prompts seem to make this more likely, and it seems to happen at random.

I can send the prompt that causes the first case, but it happens like 1 out of 300 times or something akin.

[giladgd]

@onzag If you encounter this at random throughout the day, could you please force node-llama-cpp to use Vulkan instead of CUDA (via getLlama({gpu: "vulkan"})) and report whether the issue persists?
I want to try and separate potential backends issues from higher level ones.

Also, could you please try using the quants that I made and see if you still have this issue with them?

I’ll run some tests later today to try replicating this.

[onzag]

@onzag If you encounter this at random throughout the day, could you please force node-llama-cpp to use Vulkan instead of CUDA (via getLlama({gpu: "vulkan"})) and report whether the issue persists? I want to try and separate potential backends issues from higher level ones.

Also, could you please try using the quants that I made and see if you still have this issue with them?

I’ll run some tests later today to try replicating this.

google-deepmind/gemma#622

This appears to be relevant, they are mentioning the exact words that repeat for me and enter that death loop, eg. they mention "own"...

I also have a very long prompt, DRY disabled, trying differenting sampling parameters seem to affect it but you eventually hit that.

I will try your quants, downloading.

Meanwhile I will test with vulkan.

Eh nevermind [node-llama-cpp] alloc_tensor_range: failed to allocate Vulkan0 buffer of size 396361728 I don't know why.

[giladgd]

@onzag I tried replicating this issue but it didn't reproduce in any of my tests.
From the issue you linked it seems like it may be an issue with the model weights themselves, so it's not something that could/should be solved by node-llama-cpp/llama.cpp.
Tuning the repeat penalty may help, but DRY could be better as it would force the model to not repeat itself (as opposed to repeat penalty only lowering the chance of that).