When attn_implementation = 'eager', some of the logits values become NaN.

[top-yun]

I was analyzing a task that involves both audio and visual inputs.

In order to analyze the attention map, I enabled the output_attentions = True option. However, the last token ( corresponding to assistant\n in the prompt <|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<video>\nIs the man making sound in the audio?<|im_end|>\n<|im_start|>assistant\n ) turns into NaN.

After checking, I noticed that enabling output_attentions = True forces attn_implementation to become 'eager'. Indeed, explicitly setting attn_implementation = 'eager' produces the same issue.

Is there any known fix or information about this bug?

After further investigation, I observed that the attention maps remain normal up to the 27th layer, but at the 28th (final) layer, NaN values appear in the attention map.

[top-yun]

I think I may have found the reason. The model is loaded in float16, and when using the eager implementation, some computations appear to be incorrect. After converting to bfloat16 and testing, it seems to produce the correct results.