AttributeError: 'RoFormerModel' object has no attribute 'create_extended_attention_mask_for_decoder'

[Chan-0312]

AttributeError: 'RoFormerModel' object has no attribute 'create_extended_attention_mask_for_decoder'

[JunnYu]

现在新的transformers版本应该修改了这个api，建议降低一下transformers的版本

[guofei1989]

在源码的RoFormerModel下添加如下方法即可：

def create_extended_attention_mask_for_decoder(self, input_shape, attention_mask, device):
    batch_size, seq_length = input_shape
    device = attention_mask.device
    seq_ids = torch.arange(seq_length, device=device)
    causal_mask = seq_ids[None, None, :].repeat(batch_size, seq_length, 1) <= seq_ids[None, :, None]
    # in case past_key_values are used we need to add a prefix ones mask to the causal mask
    # causal and attention masks must have same type with pytorch version < 1.3
    causal_mask = causal_mask.to(attention_mask.dtype)

    if causal_mask.shape[1] < attention_mask.shape[1]:
        prefix_seq_len = attention_mask.shape[1] - causal_mask.shape[1]
        causal_mask = torch.cat(
            [
                torch.ones((batch_size, seq_length, prefix_seq_len), device=device, dtype=causal_mask.dtype),
                causal_mask,
            ],
            axis=-1,
        )

    extended_attention_mask = causal_mask[:, None, :, :] * attention_mask[:, None, None, :]
    return extended_attention_mask