Why is padding mask defined but not used?

[hzc363]

Hello, you all Keras gurus!

I am trying to learn transformer by using this keras tutorial: https://github.com/keras-team/keras-io/blob/master/examples/nlp/neural_machine_translation_with_transformer.py.

The TransformerEncoder class allow users to add padding masks (line 260-263):
def call(self, inputs, mask=None): if mask is not None: padding_mask = ops.cast(mask[:, None, :], dtype="int32") else: padding_mask = None

But it seems that this functionality is not used when training the model (line 404):
encoder_inputs = keras.Input(shape=(None,), dtype="int64", name="encoder_inputs") x = PositionalEmbedding(sequence_length, vocab_size, embed_dim)(encoder_inputs) encoder_outputs = TransformerEncoder(embed_dim, latent_dim, num_heads)(x) encoder = keras.Model(encoder_inputs, encoder_outputs)
Since I am relatively new to this field. I am not sure if I understand the code correctly. Is it ture that the code impletmented padding mask but did not use it during the training? Why the tutorial is designed this way?

Thank you very much for the answers.

[Kxrma47]

The mask is used implicitly. PositionalEmbedding in that example creates its token embedding with mask_zero=True and implements compute_mask(). Keras attaches that mask to the symbolic tensor x and automatically passes it to a downstream layer whose call() accepts a mask argument.

So this:

x = PositionalEmbedding(...)(encoder_inputs)
encoder_outputs = TransformerEncoder(...)(x)

is effectively supplying the padding mask even though the call site does not spell out mask=.... Inside TransformerEncoder.call, the received shape is [batch, sequence]; mask[:, None, :] makes it broadcastable for attention.

You can verify the propagation directly:

embedding = PositionalEmbedding(sequence_length, vocab_size, embed_dim)
x = embedding(encoder_inputs)
print(embedding.compute_mask(encoder_inputs))

The important condition is that padding tokens are represented by zero. If you replace PositionalEmbedding, transform the sequence through a layer that destroys masks, or use a nonzero padding ID, then you must implement/propagate the mask yourself or pass an explicit attention mask.