Add the Transformer foundations course milestone

Add four tested interactive Transformer foundation lessons, exercises, course integration, and browser coverage for the approved AI from First Principles v0.2 roadmap.

Author: Doppp

README.md

@@ -9,9 +9,9 @@ database, analytics, hosted executor, or LMS.
 
 The reference course is
 [AI from First Principles](examples/ai-from-first-principles). Its current
-eight-lesson foundation covers gradient descent, backpropagation, linear
-layers, losses and optimisers, BPE, self-attention, sampling, and evaluation
-leakage.
+twelve-lesson foundation builds from gradients and linear layers through
+embeddings, multi-head attention, a Transformer block, next-token training,
+sampling, and evaluation.
 
 ## Quick start
 

apps/site/src/main.tsx

@@ -125,7 +125,7 @@ function App() {
 
         <section className="section course-feature" id="course">
           <div>
-            <p className="kicker">Course 001 · eight foundation lessons</p>
+            <p className="kicker">Course 001 · twelve foundation lessons</p>
             <h2>
               AI from
               <br />
@@ -149,9 +149,13 @@ function App() {
               ["03", "Linear layers", "Vectors, matrices, shapes, and projections"],
               ["04", "Losses and optimisers", "Cross-entropy and parameter updates"],
               ["05", "BPE tokenisation", "Pair counts and merge rules"],
-              ["06", "Self-attention", "Queries, keys, masks, and weights"],
-              ["07", "Sampling", "Temperature, top-k, and top-p"],
-              ["08", "Evaluation leakage", "Splits, contamination, and trust"],
+              ["06", "Embeddings and position", "Lookup tables and RoPE intuition"],
+              ["07", "Self-attention", "Queries, keys, masks, and weights"],
+              ["08", "Multi-head attention", "Independent mixtures and projections"],
+              ["09", "Transformer block", "RMSNorm, residuals, and SwiGLU"],
+              ["10", "Next-token training", "Shifted targets, loss, and updates"],
+              ["12", "Sampling", "Temperature, top-k, and top-p"],
+              ["13", "Evaluation leakage", "Splits, contamination, and trust"],
             ].map(([number, title, detail]) => (
               <li key={number}>
                 <span>{number}</span>

docs/implementation-status.md

@@ -28,14 +28,17 @@ Updated: 28 July 2026
 - Implemented the first v0.2 foundation milestone: shape-aware linear
   projections, stable cross-entropy, clipping, SGD, momentum, AdamW, two
   interactive lessons, and two protected exercises.
+- Implemented the second v0.2 foundation increment: embedding lookup,
+  RoPE-style positional rotation, causal multi-head attention, a pre-norm
+  residual/SwiGLU block, next-token target alignment, four interactive lessons,
+  and four protected exercises.
 
 ## Current work
 
-The v0.1 runtime MVP remains verified. The first `AI from First Principles`
-v0.2 expansion milestone is implemented and verified. The next course
-milestone adds embeddings, positional information, multi-head attention, the
-complete Transformer block, and next-token training. External DNS and
-learner-study evidence also remain.
+The v0.1 runtime MVP remains verified. The first two `AI from First Principles`
+v0.2 implementation increments are complete. The next course increment adds
+autoregressive inference, KV caching, and the small Transformer capstone.
+External DNS and learner-study evidence also remain.
 
 ## Decisions
 
@@ -70,14 +73,11 @@ npm view typescript@7 version          7.0.2
 npm view pnpm@11 version               latest 11.15.1
 official Node release status           Node 24 is LTS
 official Claude Desktop preview docs   launch schema version 0.0.1 confirmed
-pnpm check                              pass (format, lint, TS7, 22 unit tests, validators)
-pnpm course:test                        pass (6 model suites; each starter fails/reference passes)
+pnpm check                              pass (format, lint, TS7, 18 suites/51 tests, validators)
+pnpm course:test                        pass (12 model suites/38 tests; 12 starter/reference pairs)
 pnpm build                              pass (9 package entries, both courses, static site)
-pnpm test:browser                       pass (sandbox interaction, navigation, axe, 720px)
+pnpm test:browser                       pass (4 tests; sandbox, Transformer failures, axe, 720px)
 pnpm site:test                          pass (content, no-tracking copy, axe, 720px)
-foundation model tests                  pass (8 suites, 21 tests)
-foundation exercise verification        pass (8 starters fail, 8 references pass)
-foundation browser tests                pass (3 tests, new failure and training states)
 generated course smoke                  pass (validate, 2 tests, static build)
 pnpm audit --audit-level high           pass (no known vulnerabilities)
 Pages workflow YAML parse               pass

examples/ai-from-first-principles/COURSE.md

@@ -1,7 +1,7 @@
 ---
 id: ai-from-first-principles
 title: AI from First Principles
-version: 0.2.0-foundations.1
+version: 0.2.0-transformer.1
 summary: See, implement, and debug the foundations behind modern language models.
 license: CC-BY-4.0
 audience:
@@ -10,7 +10,7 @@ audience:
 prerequisites:
   - basic TypeScript
   - algebra and arrays
-estimatedHours: 8
+estimatedHours: 12
 repository: https://github.com/Doppp/explorables
 language: en
 tags:
@@ -30,6 +30,10 @@ tests. The coding agent is your tutor and debugger, not your substitute.
 3. [Vectors, matrices, and linear layers](lessons/03-vectors-matrices-linear-layers.md)
 4. [Losses and optimisers](lessons/04-losses-optimisers.md)
 5. [BPE tokenisation](lessons/05-bpe-tokenisation.md)
-6. [Self-attention](lessons/07-self-attention.md)
-7. [Sampling and generation](lessons/12-sampling.md)
-8. [Evaluation leakage](lessons/13-evaluation-leakage.md)
+6. [Embeddings and positional information](lessons/06-embeddings-positional-information.md)
+7. [Self-attention](lessons/07-self-attention.md)
+8. [Multi-head attention](lessons/08-multi-head-attention.md)
+9. [The Transformer block](lessons/09-transformer-block.md)
+10. [Next-token training](lessons/10-next-token-training.md)
+11. [Sampling and generation](lessons/12-sampling.md)
+12. [Evaluation leakage](lessons/13-evaluation-leakage.md)

examples/ai-from-first-principles/README.md

@@ -1,14 +1,14 @@
 # AI from First Principles
 
 An expanding interactive course for software developers who use AI tools and
-want to understand the machinery underneath them. The current foundation
-milestone contains eight lessons, including a linear classifier built from
-matrix multiplication, cross-entropy, and optimiser updates.
+want to understand the machinery underneath them. The current twelve-lesson
+course builds from scalar gradients through embeddings, multi-head attention,
+a complete Transformer block, and next-token training.
 
 ## Prerequisites
 
 Basic TypeScript, algebra, arrays, a terminal, Git, and the ability to read a
-test failure. The current course takes roughly eight hours.
+test failure. The current course takes roughly twelve hours.
 
 ## Start
 

examples/ai-from-first-principles/exercises/embeddings-position/README.md

@@ -0,0 +1,9 @@
+# Implement embedding lookup and pair rotation
+
+Complete `embeddingLookup` and `rotatePairs`. Return independent copies of the
+selected embedding rows, then apply a position-dependent rotation to each
+coordinate pair. The starter returns aliases into the table and adds the
+position as a scalar offset.
+
+After the tests pass, explain why rotation preserves vector norm and why actual
+RoPE applies these rotations to queries and keys.

examples/ai-from-first-principles/exercises/embeddings-position/exercise.json

@@ -0,0 +1,10 @@
+{
+  "id": "embedding-lookup-rope",
+  "title": "Implement embedding lookup and pair rotation",
+  "language": "typescript",
+  "starter": "starter",
+  "testCommand": "pnpm exec vitest run exercises/embeddings-position/tests --config vitest.exercise.config.ts",
+  "estimatedMinutes": 40,
+  "centralFiles": ["starter/embeddings.ts"],
+  "protectedPaths": ["solution"]
+}

examples/ai-from-first-principles/exercises/embeddings-position/solution/embeddings.ts

@@ -0,0 +1,51 @@
+function assertFinite(values: number[], name: string): void {
+  if (values.some((value) => !Number.isFinite(value))) {
+    throw new RangeError(`${name} must contain only finite values`);
+  }
+}
+
+export function embeddingLookup(table: number[][], tokenIds: number[]): number[][] {
+  if (table.length === 0) throw new RangeError("table must not be empty");
+  const width = table[0]?.length ?? 0;
+  if (
+    width === 0 ||
+    table.some((row) => row.length !== width) ||
+    table.some((row) => row.some((value) => !Number.isFinite(value)))
+  ) {
+    throw new RangeError("table must be rectangular and finite");
+  }
+  return tokenIds.map((tokenId) => {
+    if (!Number.isInteger(tokenId) || tokenId < 0 || tokenId >= table.length) {
+      throw new RangeError("token id is outside the table");
+    }
+    return [...(table[tokenId] ?? [])];
+  });
+}
+
+export function rotatePairs(
+  vector: number[],
+  position: number,
+  base = 10_000,
+): number[] {
+  if (vector.length === 0 || vector.length % 2 !== 0) {
+    throw new RangeError("vector width must be positive and even");
+  }
+  if (!Number.isInteger(position) || position < 0) {
+    throw new RangeError("position must be a non-negative integer");
+  }
+  if (!(base > 1) || !Number.isFinite(base)) {
+    throw new RangeError("base must be finite and greater than one");
+  }
+  assertFinite(vector, "vector");
+  const result: number[] = [];
+  for (let offset = 0; offset < vector.length; offset += 2) {
+    const pair = offset / 2;
+    const angle = position * base ** (-(2 * pair) / vector.length);
+    const cosine = Math.cos(angle);
+    const sine = Math.sin(angle);
+    const first = vector[offset] ?? 0;
+    const second = vector[offset + 1] ?? 0;
+    result.push(first * cosine - second * sine, first * sine + second * cosine);
+  }
+  return result;
+}

examples/ai-from-first-principles/exercises/embeddings-position/starter/embeddings.ts

@@ -0,0 +1,7 @@
+export function embeddingLookup(table: number[][], tokenIds: number[]): number[][] {
+  return tokenIds.map((tokenId) => table[tokenId] ?? []);
+}
+
+export function rotatePairs(vector: number[], position: number): number[] {
+  return vector.map((value) => value + position);
+}

examples/ai-from-first-principles/exercises/embeddings-position/tests/embeddings.test.ts

@@ -0,0 +1,39 @@
+import { describe, expect, it } from "vitest";
+
+const { embeddingLookup, rotatePairs } = await import(
+  process.env.EXPLORABLES_SOLUTION === "1"
+    ? "../solution/embeddings.ts"
+    : "../starter/embeddings.ts"
+);
+
+describe("embedding lookup and pair rotation", () => {
+  it("looks up rows without aliasing the table", () => {
+    const table = [
+      [1, 0],
+      [0, 1],
+    ];
+    const result = embeddingLookup(table, [1, 0]);
+    expect(result).toEqual([
+      [0, 1],
+      [1, 0],
+    ]);
+    result[0]?.splice(0, 1, 9);
+    expect(table[1]).toEqual([0, 1]);
+  });
+
+  it("leaves position zero unchanged", () => {
+    expect(rotatePairs([1, 2, 3, 4], 0)).toEqual([1, 2, 3, 4]);
+  });
+
+  it("preserves norm while changing direction", () => {
+    const vector = [1, 2, -0.5, 0.25];
+    const rotated = rotatePairs(vector, 7);
+    expect(rotated).not.toEqual(vector);
+    expect(Math.hypot(...rotated)).toBeCloseTo(Math.hypot(...vector));
+  });
+
+  it("validates identifiers and even vector width", () => {
+    expect(() => embeddingLookup([[1, 0]], [1])).toThrow(/outside/i);
+    expect(() => rotatePairs([1, 2, 3], 1)).toThrow(/even/i);
+  });
+});