A linear model can only draw a straight line. Stack two linear layers with a non-linearity between them and you get something that can curve, twist, and approximate (in theory) any continuous function.
nn.Module— the base class for every model. Holds parameters, supports.to(device),.train(),.eval(), etc.nn.Sequential— sugar for "apply these layers in order".- ReLU —
max(0, x). The most common activation. Cheap and works well. CrossEntropyLoss— for multi-class classification. Combines log-softmax + NLL.- Adam — an optimizer that adapts the learning rate per-parameter. Usually a good default.
Two linear layers without an activation collapse into a single linear layer (matrix multiplication is associative). The non-linearity in between is what gives the model expressive power.
- Remove the
nn.ReLU()lines — accuracy collapses to a logistic-regression baseline. - Shrink
hiddento4— under-fitting. - Train for 3000 epochs — over-fitting (loss keeps dropping but the model memorizes noise).
python 05_mlp/main.py