Describe the bug
DeepSpeed accepts a negative maximum-output-token limit during configuration parsing and startup instead of rejecting the invalid value.
Steps to reproduce the behavior:
- Save the following as a shell command and run it in a shell on a machine with a free CUDA device.
- Run the command in the target Python environment with the listed dependencies installed.
- Observe the startup or generation result.
import torch
import deepspeed
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "/root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-R1-Distill-Llama-8B/snapshots/6a6f4aa4197940add57724a7707d069478df56b1"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
model_name, torch_dtype=torch.bfloat16
).to("cuda")
engine = deepspeed.init_inference(
model, config={"max_out_tokens": -1}
)
inputs = tokenizer("Reply with exactly OK.", return_tensors="pt")
device = next(engine.module.parameters()).device
inputs = {key: value.to(device) for key, value in inputs.items()}
with torch.no_grad():
output = engine.generate(**inputs, max_new_tokens=8, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Expected behavior
A clear and concise description of what you expected to happen.
Actual behavior
The generation request fails with: Input with size 6 exceeds maximum length of -1. Please increase max_tokens in the DeepSpeed Inference Config.
Expected behavior
Values below the valid domain should be rejected during configuration validation before model initialization or request execution.
System info (please complete the following information):
- OS: Ubuntu 22.04.3 LTS
- GPU count and types: 1 x NVIDIA H100 80GB HBM3
- DeepSpeed version: 0.19.1
- DeepSpeed-MII: not used
- PyTorch: 2.9.1+cu128
- Transformers: 5.15.0
- CUDA reported by PyTorch: 12.8
- nvcc: 13.3
- Python version: 3.12.12
- Target runtime: /root/anaconda3/envs/deepspeed128
Docker context
No Docker image was used. The reproduction ran directly on Ubuntu 22.04.3 LTS in the target conda environment.
Describe the bug
DeepSpeed accepts a negative maximum-output-token limit during configuration parsing and startup instead of rejecting the invalid value.
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Actual behavior
The generation request fails with: Input with size 6 exceeds maximum length of -1. Please increase max_tokens in the DeepSpeed Inference Config.
Expected behavior
Values below the valid domain should be rejected during configuration validation before model initialization or request execution.
System info (please complete the following information):
Docker context
No Docker image was used. The reproduction ran directly on Ubuntu 22.04.3 LTS in the target conda environment.