When I run the GRPO demo, I find that format_reward is always 0！！！

[asirgogogo]

ACCELERATE_LOG_LEVEL=info accelerate launch --config_file recipes/accelerate_configs/zero2.yaml \
    --num_processes=7 src/open_r1/grpo.py \
    --config recipes/Qwen2.5-Math-7B/grpo/config_simple_rl.yaml

I'm sure that I didn't modify the parameters in “config_simple_rl.yaml”.

[tpoisonooo]

format_reward is exactly matching “<think>.. <answer>” format, then return 0 or 1.

I guess:

• this reward_func curve is too sharp
• the training dataset (NuminaMath-TIR) does not have this response format.

maybe a good PR ?

[tpoisonooo]

def format_reward(completions, **kwargs):
    """Reward function that checks if the completion has a specific format."""
    pattern = r"^<think>.*?</think><answer>.*?</answer>$"
    completion_contents = [completion[0]["content"] for completion in completions]
    matches = [re.match(pattern, content) for content in completion_contents]
    return [1.0 if match else 0.0 for match in matches]

[ytchx1999]

same

[hellen9527]

I want to ask how big a machine is needed to train a 7B model. I have 8 L20 GPUs, and when I train for 10 steps, the loss is 0, and the format_reward is also 0, though the accuracy_reward is not 0. I’m not sure why. However, at step 15, I get an error saying that the GPU memory is insufficient. Could you let me know what machine you're using?

[asirgogogo]

@hellen9527 I use 8xA800 GPUs

[asirgogogo]

@tpoisonooo According to the settings, “DigitalLearningGmbH/MATH-lighteval” dataset is used.

[hellen9527]

可能是有换行符导致正则表达式判断为错了，返回0分，另外一个问题里说用这个格式分换下面的正则表达式效果更好

pattern = r"<think>.*?</think>\s*<answer>.*?</answer>"
completion_contents = [completion[0]["content"] for completion in completions]
matches = [re.match(pattern, content, re.DOTALL) for content in completion_contents]

[tpoisonooo]

可能是有换行符导致正则表达式判断为错了，返回0分，另外一个问题里说用这个格式分换下面的正则表达式效果更好

pattern = r"<think>.*?</think>\s*<answer>.*?</answer>"
completion_contents = [completion[0]["content"] for completion in completions]
matches = [re.match(pattern, content, re.DOTALL) for content in completion_contents]

下面的 expr 没卡那么严，确实能缓解。

[brishtiteveja]

Related node: Please also check whether you are using cuda 12.4 (as it is shown as Caution in ReadMe). I was initially using cuda 12.1 and then switched to 12.4

[izlley]

The issue has been fixed. #238

[Moyhub]

@izlley After I modified it by pressing #238, the forwar reward was still almost zero when I used qwen2.5-math-7B for GRPO training, as follows:

[ZhenweiAn]

Perhaps there should have a cold start for fomat before training grpo

[yuleiqin]

same here, with the new format reward, the model still fails to deliever correct formats

def format_reward(completions, **kwargs): """Reward function that checks if the completion has a specific format.""" pattern = r"^<think>.*?</think>\s*<answer>.*?</answer>$" completion_contents = [completion[0]["content"] for completion in completions] matches = [re.match(pattern, content, re.DOTALL | re.MULTILINE) for content in completion_contents] return [1.0 if match else 0.0 for match in matches]

8 H20 GPUs, DeepSeek-R1-Distill-Qwen-7B-GRPO

@izlley After I modified it by pressing #238, the forwar reward was still almost zero when I used qwen2.5-math-7B for GRPO training, as follows:

[yihao-123]

I use the Qwen2.5-7B-Base， and it can output the correct format. However, the Qwen2.5-7B-Math-Instruct cannot output the correct format.