You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
| $\mathsf{Data} \cap \mathsf{Algorithm} \cap \mathsf{Machine}$ | ML Systems Engineering | iron law, Roofline, training loops, serving |@sec-model-training, @sec-model-serving, @sec-benchmarking|
71
71
72
-
: **D·A·M Intersection Reference**: Each zone maps specific techniques to the axes they span and the chapters that cover them. The pairwise intersections require reasoning about two domains simultaneously; the center requires all three. {#tbl-dam-intersections tbl-colwidths="[12,25,36,27]"}
72
+
: **D·A·M Intersection Reference**: Each zone maps specific techniques to the axes they span and the chapters that cover them. The pairwise intersections require reasoning about two domains simultaneously; the center requires all three. {#tbl-dam-intersections tbl-colwidths="[20,23,30,27]"}
73
73
74
74
The pure zones contain concepts that belong entirely to one axis: storage formats and distribution properties are purely Data concerns, loss functions and gradient computations are purely Algorithm, and silicon physics and peak FLOP/s are purely Machine. These are the topics where single-domain expertise suffices.
75
75
@@ -108,7 +108,7 @@ The systems engineer's job is to make these components run in parallel, not in s
108
108
|**Async Gradient Sync**| M (communication) overlaps A | Overlap bucketed AllReduce with remaining backward computation |
109
109
|**Double Buffering**| D overlaps M | Fill buffer N+1 while computing on buffer N |
110
110
111
-
: **D·A·M Overlap Techniques**: Each technique allows one D·A·M axis to execute while another is in flight, converting the iron law's additive terms into overlapped terms. The payoff is transforming $T = a + b$ into $T = \max(a, b)$, which can cut latency nearly in half when the terms are balanced. {#tbl-dam-overlap}
111
+
: **D·A·M Overlap Techniques**: Each technique allows one D·A·M axis to execute while another is in flight, converting the iron law's additive terms into overlapped terms. The payoff is transforming $T = a + b$ into $T = \max(a, b)$, which can cut latency nearly in half when the terms are balanced. {#tbl-dam-overlap tbl-colwidths="[22,32,46]"}
112
112
113
113
Overlap only helps when the D·A·M axes are reasonably balanced. If one term dominates (for example, severely memory bound), overlapping the smaller term with the larger yields negligible gain—the max is still dominated by the same bottleneck. Overlap provides the greatest benefit when $D_{\text{vol}}/\text{BW} \approx O/(R_{\text{peak}} \cdot \eta_{\text{hw}})$. The latency term is the important exception.
114
114
@@ -147,12 +147,12 @@ Common industry labels map to D·A·M components as follows: memory bound typica
147
147
Once the bottleneck is identified, @tbl-bottleneck-actions shows which optimizations help and which ones are wasted:
148
148
149
149
|**If the workload is...**|**Dominant Term**|**Optimization That Works**|**Optimization That is Wasted**|
: **What Works vs. What Is Wasted**: Optimizing the wrong term yields exactly zero improvement. A memory-bound large language model (LLM) will not speed up from a faster accelerator; the accelerator will simply idle faster while waiting for memory. {#tbl-bottleneck-actions tbl-colwidths="[17,17,33,33]"}
155
+
: **What Works vs. What Is Wasted**: Optimizing the wrong term yields exactly zero improvement. A memory-bound large language model (LLM) will not speed up from a faster accelerator; the accelerator will simply idle faster while waiting for memory. {#tbl-bottleneck-actions tbl-colwidths="[19,17,32,32]"}
156
156
157
157
Knowing what works also means recognizing what does not. In practice, teams under deadline pressure repeatedly fall into the same traps—optimizing the wrong axis with confidence. These failure modes are common enough to deserve their own names.
158
158
@@ -269,7 +269,7 @@ To move beyond qualitative guessing, the efficiency ratios in @tbl-dam-scorecard
269
269
[^fn-mfu-scorecard]: **MFU (Model FLOPs Utilization)**: The ratio of achieved model FLOP/s to the hardware's theoretical peak FLOP/s, introduced in the PaLM paper [@chowdhery2022palm]. Unlike raw accelerator utilization (which counts any work the accelerator performs), MFU measures only *useful* model computation rate, excluding overhead like gradient synchronization and memory management. @sec-benchmarking covers MFU in depth.
0 commit comments