You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
| Master + Adam states per parameter (FP32) |`{python} MemoryConstants.BYTES_ADAM_STATE_val_str`|`{python} MemoryConstants.BYTES_ADAM_STATE_unit_str`|
493
+
| Master + Adam states per parameter (FP32) |`{python} MemoryConstants.MASTER_PLUS_ADAM_val_str`|`{python} MemoryConstants.MASTER_PLUS_ADAM_unit_str`|
494
494
| Bytes per parameter (mixed-precision Adam) |`{python} MemoryConstants.ADAM_MIXED_PREC_TOTAL_val_str`|`{python} MemoryConstants.ADAM_MIXED_PREC_TOTAL_unit_str`|
495
495
496
496
: **Training Memory Conventions**: Per-parameter storage for mixed-precision Adam (2 + 2 + 12 = 16 bytes before activations). Book convention for napkin math; see [@nvidia_tensors_fp16_2017] for mixed-precision training context. {#tbl-assumptions-training-memory tbl-colwidths="[50,20,30]"}
These numbers are representable in FP64 but overflow FP32 (max $\approx 3.4 \times 10^{38}$). With FP16 (max $\approx 65{,}504$), even $e^{12}$ overflows. In practice, logits of magnitude 100 are not unusual in deep networks, and FP16/BF16 is the standard training precision—so naive softmax fails routinely.
757
+
These numbers are representable in FP64 but overflow FP32 (max $\approx 3.4 \times 10^{38}$). With FP16 (max $\approx 65{,}504$), even $e^{12}$ overflows. Raw logits rarely reach magnitude 100, but large intermediate values do arise (for example, unscaled attention scores or overconfident outputs), and because FP16 and BF16 are the standard training precisions, a naive softmax risks overflow often enough that the stable form is used by default.
758
758
759
759
**With the trick**: Subtract $a = \max(z) =$ `{python} LogSumExp.a_str`:
Copy file name to clipboardExpand all lines: book/quarto/contents/vol1/backmatter/appendix_machine.qmd
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -1237,7 +1237,7 @@ With the full toolkit assembled---reference numbers, performance models, archite
1237
1237
1238
1238
::: {.callout-takeaways title="Numbers every engineer should know"}
1239
1239
1240
-
***Energy dominates**: In the cited 45 nm reference, a 32-bit DRAM read uses roughly 580$\times$ the energy of one FP16 multiply. The ratio varies by platform, but it makes arithmetic intensity and reuse central design metrics.
1240
+
***Energy dominates**: In the cited 45 nm reference, a 32-bit DRAM read uses roughly `{python} NumbersToKnow.dram_vs_compute_ratio_mult_str` the energy of one FP16 multiply. The ratio varies by platform, but it makes arithmetic intensity and reuse central design metrics.
1241
1241
***The Roofline model**: The Roofline Model shows whether compute throughput or memory bandwidth is the active ceiling. Batch-1 weight-streaming inference is often below the ridge; batching can raise intensity, but shape, precision, reuse, and the ridge point determine whether it crosses.
1242
1242
***Amdahl's Law**: Amdahl's Law caps strong-scaling speedup at $1/s$ (where $s$ is the serial fraction). Gustafson's Law shows that scaling the problem alongside hardware yields near-linear throughput gains—the paradigm that makes large-scale training feasible.
1243
1243
***Little's Law**: $Q_{\text{req}} = \lambda_{\text{arr}} T_{\text{lat}}$ links mean population, throughput, and time in system; memory also depends on where queued and executing state resides.
***Machine Learning Benchmarking***\index{Benchmarking!definition} is the empirical measurement of a system's end-to-end performance on representative ML workloads, designed to decouple marketed peak specifications from the sustained throughput and latency achievable under realistic operating conditions.
102
107
103
-
1.**Significance**: The gap between peak and sustained performance is large and structurally unavoidable. An A100 GPU delivers `{python} A100PeakBenchmark.a100_fp16_tflop_per_s_str` (BF16) at peak, but production transformer training runs typically sustain `{python} A100PeakBenchmark.achievable_base_tflop_str`--`{python} A100PeakBenchmark.achievable_tuned_tflop_str` (`{python} A100PeakBenchmark.mfu_base_pct_str`--`{python} A100PeakBenchmark.mfu_tuned_pct_str` MFU), about a 2--3.5$\times$ gap that exists even in optimally tuned systems due to memory stalls, pipeline bubbles, and kernel launch overhead. Benchmarking quantifies this $\eta_{\text{hw}}$ gap; vendor spec sheets do not.
108
+
1.**Significance**: The gap between peak and sustained performance is large and structurally unavoidable. An A100 GPU delivers `{python} A100PeakBenchmark.a100_fp16_tflop_per_s_str` (BF16) at peak, but production transformer training runs typically sustain `{python} A100PeakBenchmark.achievable_base_tflop_str`--`{python} A100PeakBenchmark.achievable_tuned_tflop_str` (`{python} A100PeakBenchmark.mfu_base_pct_str`--`{python} A100PeakBenchmark.mfu_tuned_pct_str` MFU), about a `{python} A100PeakBenchmark.sustained_gap_range_mult_str` gap that exists even in optimally tuned systems due to memory stalls, pipeline bubbles, and kernel launch overhead. Benchmarking quantifies this $\eta_{\text{hw}}$ gap; vendor spec sheets do not.
104
109
2.**Distinction**: Unlike micro-benchmarks, which measure individual kernel performance such as a general matrix multiply (GEMM) at peak matrix dimensions, ML benchmarks measure the full stack: data loading, preprocessing, forward pass, gradient computation, optimizer step, and checkpoint I/O—exposing bottlenecks that individual-component benchmarks will never reveal.
105
110
3.**Common pitfall**: A frequent misconception is that benchmark numbers are stable references. Both the workload (new model architectures) and the hardware (new GPU generations) evolve, so a result that leads a benchmark under one version often becomes the baseline under a later version, making year-over-year comparisons meaningful only when the benchmark version is held constant.
106
111
@@ -4127,7 +4132,7 @@ The final category of data benchmarking asks whether models will generalize from
4127
4132
4128
4133
The standard assumption underlying held-out evaluation, that test data comes from the same distribution as training data, is routinely violated in practice. Test sets constructed years after training data may reflect distribution drift as the world changes. Test sets from different geographic regions may reflect population shift. A model with strong held-out accuracy can drop sharply when deployed to a region or time period the test set did not represent. Standard held-out evaluation overestimates deployment performance whenever the i.i.d. (independent and identically distributed) assumption fails.
4129
4134
4130
-
The true test is train-to-production alignment, and this is far harder to measure because production data differs from training data in ways that held-out test sets often fail to capture. Production images come from different cameras with different characteristics. Production users come from different populations with different behaviors. Production inputs include edge cases that curated test sets systematically exclude. The WILDS[^fn-wilds] benchmark [@koh2021wilds] was designed specifically to evaluate models under realistic distribution shifts: hospital systems with different patient populations, wildlife cameras at different locations, satellite imagery from different time periods. The results reveal a stark reality: models achieving 90 percent+ accuracy on in-distribution test sets may drop to 60 percent under these realistic shifts.
4135
+
The true test is train-to-production alignment, and this is far harder to measure because production data differs from training data in ways that held-out test sets often fail to capture. Production images come from different cameras with different characteristics. Production users come from different populations with different behaviors. Production inputs include edge cases that curated test sets systematically exclude. The WILDS[^fn-wilds] benchmark [@koh2021wilds] was designed specifically to evaluate models under realistic distribution shifts: hospital systems with different patient populations, wildlife cameras at different locations, satellite imagery from different time periods. The results reveal a stark reality: models achieving 97 percent accuracy on in-distribution test sets may drop to 70 percent under these realistic shifts.
4131
4136
4132
4137
[^fn-wilds]: **WILDS**: Stanford's 2021 benchmark of ten datasets with real-world distribution shifts: hospital patient population changes (Camelyon17), wildlife camera location shifts (iWildCam), and satellite imagery temporal drift (PovertyMap). WILDS quantifies the deployment gap: models achieving 97 percent in-distribution accuracy can drop to 70 percent under these realistic shifts, demonstrating that standard held-out evaluation systematically overestimates production performance when the i.i.d. assumption\index{I.I.D. Assumption!held-out evaluation limitation} fails. \index{WILDS!distribution shift benchmark}
Data-centric AI\index{Data-Centric AI!dataset enhancement paradigm} reflects an important shift in understanding that challenges the "more data is always better" assumption: *better* datasets, not just *larger* ones, produce more reliable and generalizable AI systems. Initiatives like DataPerf\index{DataPerf!data quality benchmark} [@mazumder2024dataperf] and DataComp\index{DataComp!data-centric benchmark}[^fn-datacomp] have emerged to systematically evaluate how dataset improvements affect model performance. For instance, DataComp [@beaumont2023] demonstrated that models trained on a carefully curated 30 percent subset of data achieved better results than those trained on the complete dataset, challenging the assumption that more data automatically leads to better performance.
4289
4294
4290
-
[^fn-datacomp]: **DataComp**: Introduced in 2023, DataComp inverts the standard benchmark by fixing the model and training code, letting participants compete on dataset curation alone. Results showed that a carefully filtered 30 percent subset matched models trained on 10$\times$ larger unfiltered data, quantifying a systems insight: for many workloads, engineering the data pipeline yields greater performance gains per dollar than scaling compute. \index{DataComp!data-centric benchmarking}
4295
+
[^fn-datacomp]: **DataComp**: Introduced in 2023, DataComp inverts the standard benchmark by fixing the model and training code, letting participants compete on dataset curation alone. Results showed that a carefully filtered 30 percent subset outperformed models trained on the full unfiltered pool, quantifying a systems insight: for many workloads, engineering the data pipeline yields greater performance gains per dollar than scaling compute. \index{DataComp!data-centric benchmarking}
4291
4296
4292
4297
A persistent challenge in data benchmarking emerges from dataset saturation\index{Dataset Saturation!performance capability illusion}. When models achieve near-perfect accuracy on benchmarks like ImageNet, practitioners must distinguish whether performance gains represent genuine capability advances or merely optimization to existing test sets. As the timeline in @fig-dataset-saturation illustrates, widely tracked AI benchmarks have repeatedly crossed reported human baselines, making each corresponding benchmark less useful as a differentiator [@perrault2024artificial].
Copy file name to clipboardExpand all lines: book/quarto/contents/vol1/conclusion/conclusion.qmd
+2-2Lines changed: 2 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -104,7 +104,7 @@ The five Lighthouse workloads expose distinct constraint regimes:
104
104
-**GPT-2/Llama**\index{GPT-2!memory bandwidth}: Autoregressive language generation exposes the opposite wall, where every token reloads enough state that memory bandwidth, KV-cache growth, and model parallelism dominate serving cost.
105
105
-**MobileNetV2**\index{MobileNetV2!efficiency under constraint}: Depthwise separable convolutions and INT8 quantization trade representational capacity for mobile NPU deployment in a power-constrained regime.
106
106
-**DLRM**\index{DLRM!capacity-bound workload}: Terabyte-scale embedding tables shift the binding constraint from memory *bandwidth* to memory *capacity*, forcing engineers to design around where data physically resides and how sparse operations behave.
107
-
-**Keyword spotting (KWS)/Wake Vision**\index{Keyword Spotting!sub-megabyte models}: Sub-megabyte models running on microcontrollers with always-on inference under microwatt power budgets make every byte and every milliwatt matter.
107
+
-**Keyword spotting (KWS)/Wake Vision**\index{Keyword Spotting!sub-megabyte models}: Sub-megabyte models running on microcontrollers with always-on inference under milliwatt power budgets make every byte and every milliwatt matter.
108
108
109
109
Together, these five workloads span the full deployment spectrum from data center to microcontroller, probing the bottlenecks these principles diagnose and testing every optimization strategy the book has taught. The systems thinking we developed by tracing these Lighthouses across chapters, from architecture design through training, optimization, and deployment, is the integrated perspective that distinguishes ML systems engineering from isolated algorithm development.
110
110
@@ -1058,7 +1058,7 @@ The next frontier is scale: models no longer fit on one machine, failures become
1058
1058
***Assumptions matter across implementations**: The thirteen principles turn framework-specific craft into measurable reasoning by combining physical bounds, decompositions, fitted models, requirements, and heuristics. Apply each only within its stated scope.
1059
1059
***Trace displaced costs without assuming conservation**: Compression, batching, monitoring, and governance can relocate burdens across data, algorithm, and machine, while good design can remove accidental complexity outright. Measurement distinguishes the two cases.
1060
1060
***Boundaries reveal the bottleneck**: A 70-billion-parameter Llama 2 can be about `{python} ConclusionRoofline.ratio_mult_str` memory-bound on H100, and p99 latency can sit `{python} TailLatencyRatio.conclusion_tail_ratio_mult_str` above the mean. Systems thinking means measuring where physics, traffic, and users bind.
1061
-
***Scale changes the binding term**: The next frontier is scale, where a thousand-GPU pool turns multi-year component MTTF into sub-day cluster MTBF. The physics stays, but the constraint moves to fleets.
1061
+
***Scale changes the binding term**: The next frontier is scale, where a thousand-GPU pool turns multi-year component MTTF into days-scale cluster MTBF. The physics stays, but the constraint moves to fleets.
0 commit comments