Skip to content

Fix Issue #140: GGML Assert Batch Size Failure with Large Prompts - #158

Merged
Michael-A-Kuykendall merged 1 commit into
mainfrom
fix/issue-140-ggml-assert-batch-size
Dec 8, 2025
Merged

Fix Issue #140: GGML Assert Batch Size Failure with Large Prompts#158
Michael-A-Kuykendall merged 1 commit into
mainfrom
fix/issue-140-ggml-assert-batch-size

Conversation

@Michael-A-Kuykendall

Copy link
Copy Markdown
Owner

This PR fixes the GGML assert failure when processing large prompts that exceed the hardcoded batch size limit.

Problem

Issue #140 reported GGML_ASSERT(n_tokens_all <= cparams.n_batch) failures when processing large prompts, particularly with DeepSeek-R1 models that have extensive system messages.

Root Cause

The batch size was hardcoded to 2048 tokens, causing assertion failures when context length exceeded this limit.

Solution

Implemented adaptive batch size calculation that scales based on context length:

  • Minimum batch size: 2048 tokens
  • Maximum batch size: 8192 tokens
  • Scales dynamically to ensure batch size >= context length

Changes

  • Added calculate_adaptive_batch_size() method in src/engine/llama.rs
  • Modified context creation to use adaptive batch size instead of hardcoded 2048
  • Added comprehensive regression tests in tests/regression/issue_140_ggml_assert_batch_size.rs

Testing

  • All existing tests pass (566 total)
  • New regression tests validate batch size calculation for various context lengths
  • Release gates pass: build, CUDA, packaging, binary size, test suite, documentation

Impact

This fix prevents server crashes when processing large prompts while maintaining memory efficiency through adaptive sizing.

@Michael-A-Kuykendall
Michael-A-Kuykendall merged commit 70bdc06 into main Dec 8, 2025
4 of 8 checks passed
@Michael-A-Kuykendall
Michael-A-Kuykendall deleted the fix/issue-140-ggml-assert-batch-size branch May 26, 2026 14:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant