Skip to content

[BUG] DeepSpeed accepts negative max_out_tokens and fails generation requests at runtime #8339

Description

@Redempt1onzzZZ

Describe the bug
DeepSpeed accepts a negative maximum-output-token limit during configuration parsing and startup instead of rejecting the invalid value.

Steps to reproduce the behavior:

  1. Save the following as a shell command and run it in a shell on a machine with a free CUDA device.
  2. Run the command in the target Python environment with the listed dependencies installed.
  3. Observe the startup or generation result.
import torch
import deepspeed
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "/root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-R1-Distill-Llama-8B/snapshots/6a6f4aa4197940add57724a7707d069478df56b1"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name, torch_dtype=torch.bfloat16
).to("cuda")
engine = deepspeed.init_inference(
    model, config={"max_out_tokens": -1}
)
inputs = tokenizer("Reply with exactly OK.", return_tensors="pt")
device = next(engine.module.parameters()).device
inputs = {key: value.to(device) for key, value in inputs.items()}
with torch.no_grad():
    output = engine.generate(**inputs, max_new_tokens=8, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Expected behavior
A clear and concise description of what you expected to happen.

Actual behavior

The generation request fails with: Input with size 6 exceeds maximum length of -1. Please increase max_tokens in the DeepSpeed Inference Config.

Expected behavior

Values below the valid domain should be rejected during configuration validation before model initialization or request execution.

System info (please complete the following information):

  • OS: Ubuntu 22.04.3 LTS
  • GPU count and types: 1 x NVIDIA H100 80GB HBM3
  • DeepSpeed version: 0.19.1
  • DeepSpeed-MII: not used
  • PyTorch: 2.9.1+cu128
  • Transformers: 5.15.0
  • CUDA reported by PyTorch: 12.8
  • nvcc: 13.3
  • Python version: 3.12.12
  • Target runtime: /root/anaconda3/envs/deepspeed128

Docker context

No Docker image was used. The reproduction ran directly on Ubuntu 22.04.3 LTS in the target conda environment.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions