Skip to content

Commit 7c518bc

Browse files
committed
Refactor Gemma distribution tests onto the shared helper
Depends on keras-team#2809 (adds TestCase.run_distribution_test) for tests to pass -- this PR's own diff is independent (based directly on master, not stacked) so it stays small and reviewable on its own. Refactors Gemma's existing test_distribution/test_distribution_with_lora onto the shared helper. Adds test_layout_map_mesh_shapes (CI-safe sweep across representative 2D/3D mesh shapes) and test_layout_map_live_presets (manual, Kaggle-gated sweep across the full preset registry). Gemma's get_layout_map itself is unchanged -- already correct.
1 parent 8375767 commit 7c518bc

2 files changed

Lines changed: 381 additions & 51 deletions

File tree

keras_hub/src/models/gemma/gemma_backbone.py

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -296,11 +296,22 @@ def get_layout_map(
296296
# See https://arxiv.org/abs/2403.08295
297297
layout_map = keras.distribution.LayoutMap(device_mesh)
298298
layout_map["token_embedding/embeddings"] = (model_dim, data_dim)
299-
layout_map["decoder_block.*attention.*(query|key|value).kernel"] = (
299+
# Key/value kernels are sized by num_key_value_heads (GQA/MQA),
300+
# which is independent of and typically much smaller than
301+
# num_query_heads -- sharding that small axis on the model-parallel
302+
# dim would raise an IndivisibleError whenever the device count
303+
# doesn't divide it, so they are left replicated on that axis while
304+
# query stays sharded.
305+
layout_map["decoder_block.*attention.*query.kernel"] = (
300306
model_dim,
301307
data_dim,
302308
None,
303309
)
310+
layout_map["decoder_block.*attention.*(key|value).kernel"] = (
311+
None,
312+
data_dim,
313+
None,
314+
)
304315
layout_map["decoder_block.*attention_output.kernel"] = (
305316
model_dim,
306317
None,

0 commit comments

Comments
 (0)