The English lessons are canonical. Translated lesson markdown is machine-generated and never committed to main — it lives on a separate translations branch, and the site fetches it at runtime the same free way it fetches English (raw.githubusercontent.com). No Vercel functions, no KV, no per-view cost. The one exception is the per-language README, which is hand-authored for quality and committed to main (see below).
This pipeline covers lessons under phases/ only. Certification curricula remain English-only and are intentionally excluded from both machine translation and book generation.
The default engine is NLLB-200 (No Language Left Behind, Meta AI, 2022), an open translation model covering 200 languages, which runs inside the GitHub Actions runner with no API key and no bill. Compute is the free CI runner; there is no translation-API cost at all. Paid LLM providers (Anthropic/OpenAI) and DeepL remain available as optional higher-quality upgrades via --provider.
The languages.json registry drives the machine-translation surfaces — lessons, the site switcher, and the book matrix. The per-language README is separate: its language set and translated blocks live in scripts/readme_translations.py, hand-authored for quality and independent of the registry.
| Piece | Where | Committed to main? |
|---|---|---|
| Language registry | languages.json |
Yes (one small file) |
| Lesson translation script | scripts/translate_lessons.py (prose-only walker) |
Yes |
| README generator + translations | scripts/build_readme_i18n.py + scripts/readme_translations.py (hand-authored) |
Yes |
| CI workflow | .github/workflows/translate.yml (sharded per language × phase) |
Yes |
| Translated lesson markdown | translations branch, i18n/<lang>/…/<lang>.md |
No (separate branch) |
| Translated README | i18n/<lang>/README.md, linked from the header of README.md |
Yes |
| Translated books | scripts/build_book.py --lang <code> reads the translated markdown and emits aiefs-vol{n}-{slug}-<lang>.epub/.pdf as release assets (English fallback per lesson) |
No (release assets) |
| Switcher options | site/langs.js (generated by build.js from the source/ci:true set) |
No (gitignored, like data.js) |
| Site switcher | site/lesson.html |
Yes (the UI + fetch logic) |
Add one entry to languages.json, the single source of truth. Fields:
code— the tag used in filenames, URLs, and the switcher value.name— English name, used in the LLM-provider prompt.native— label shown in the switcher.nllb— the FLORES-200 code (e.g.zho_Hans,tur_Latn); required for the default NLLB provider —translate_lessons.pyexits if a requested language has none.ci—trueopts the language into the auto push-triggered workflow's build matrix.workflow_dispatchcan request any registered language regardless.
Both matrices read from ci: true, so the same flag opts a language into lesson translation and into the site switcher — build.js only emits source/ci:true languages into site/langs.js, so the switcher never offers a language the site can't serve. The README bar is independent: it can list any language readme_translations.py covers. Nothing else to touch.
- A push to
mainthat changesphases/**/docs/en.md,languages.json, or the translation workflow itself triggers.github/workflows/translate.yml. Certification lesson changes do not trigger it. - The matrix is one job per (language, phase). A full 503-lesson language run is ~27h on a CPU runner, far past the 6-hour job limit, so a per-language job always timed out before publishing. Splitting by phase keeps the largest job (phase 19, 85 lessons) near ~4.5h, well under the limit. Each job restores its language's prior output from the branch, runs
translate_lessons.py --phase <phase>, and pushes only its owni18n/<lang>/phases/<phase>/slice back with a fetch-and-retry loop, so disjoint slices merge without collisions. - The site's language switcher sets
?lang=<code>;fetchLessonpulls…/translations/i18n/<lang>/…/<lang>.mdand falls back to canonical English if that translation isn't published yet. English (lang=en) is byte-identical to the pre-i18n fetch path.
Every lesson is keyed by the sha256 of its English source in a per-(language, phase) cache (i18n/<lang>/.cache/<phase>.json), written per lesson in the runner and published to the translations branch when the job finishes. So:
- The first run translates everything once. Every completed run after it re-translates only lessons whose English actually changed — everything else is a cache hit and is skipped instantly.
- A cancelled or interrupted phase job does not publish, so it loses that job's runner-local progress and the next run resumes that phase from the last published cache. Because each shard is a single phase, it comfortably finishes and publishes within the job limit.
The README is the landing page, so its translations are hand-authored (highest quality) rather than machine-generated, and they live in main at i18n/<lang>/README.md. scripts/readme_translations.py holds the translated strings keyed by the exact English block; scripts/build_readme_i18n.py rebuilds each file by replacing only the translated line-spans in a copy of the English README. Everything else — the banner, badges, the 584-row lesson table, all code and link text — is preserved verbatim (asserted by a round-trip identity check on every run). The one deliberate transform is on link and image targets: repo-root-relative paths get a ../../ prefix so they still resolve from two directories deep (fenced code is skipped, so code that looks like a link is left alone). Any block without a translation falls back to English, so coverage can grow language by language.
python3 scripts/build_readme_i18n.py --check runs in CI (curriculum.yml) and fails if any committed README is stale, and the README-counts bot regenerates them whenever it syncs the English stats block.
NLLB is a sentence translator, not an instruction-following model, so the walker never lets it see anything but prose. It processes the lesson line by line: fenced code blocks, tables, image lines, and the metadata header (**Type:** … **Time:**) pass through verbatim; on every other line it protects inline `code`, $math$, image refs, full markdown links, **bold** terms, and URLs behind placeholders, translates only the plain-text fragments between them, then restores. Bold spans are kept because in this course they are almost always technical terms (Skip-gram, negative sampling) that should stay English.
This is verified mechanically: running the walker with an identity "translation" reproduces all 503 lessons byte-for-byte, so a real run can only ever change prose. If a placeholder fails to round-trip, that lesson keeps its English (fail-safe).
Default provider nllb: free. NLLB-200 runs in the GitHub Actions runner — no API key, no secret, no per-token bill. The only resource is free CI compute; the model is cached between runs and only changed lessons are re-translated (sha256 cache). Vercel cost does not change.
Optional upgrades (--provider anthropic|openai|deepl) give higher fluency at low-tens-of-dollars per full pass per language, if you later want to polish a high-traffic language. Those providers read the key from the LLM_API_KEY environment variable (DeepL uses the same variable for its auth key). NLLB stays the zero-cost default.
Below is 05 · 03 Word Embeddings translated through Step 1, showing that prose reads naturally while code, math, king - man + woman, technical terms, and the metadata header survive untouched. This sample is LLM-tier fluency (what --provider anthropic produces); the free NLLB-200 default is solid and fully faithful on the preservation guarantee, with slightly plainer phrasing. Upgrade a high-traffic language to an LLM provider later if you want this level of polish.
词以其所处的语境为意。用一个浅层网络训练这个想法,几何结构便会自然浮现。
Type: Build · Languages: Python · Prerequisites: 阶段 5 · 02(BoW + TF-IDF)、阶段 3 · 03(从零实现反向传播) · Time: 约 75 分钟
问题 —— TF-IDF 知道
dog和puppy是不同的词,但不知道二者含义几乎相同。在dog上训练的分类器无法泛化到讨论puppy的评论。你可以靠列举同义词勉强弥补,但这在罕见词、领域术语,以及所有你未曾预料的语言上都会失效。你想要的是这样一种表示:
dog和puppy在空间中彼此靠近;king - man + woman落在queen附近;在dog上训练的模型能免费地把部分信号迁移到puppy。核心概念 —— 分布假说(Firth,1957):"要了解一个词,就看它与哪些词为伴。" Word2Vec 有两种形式:Skip-gram 给定中心词预测周围词(窗口为 2 时
cat -> (the, sat, on));CBOW(连续词袋) 给定周围词预测中心词。网络只有一个隐藏层且无非线性:输入是词表上的 one-hot 向量,输出是 softmax,训练后丢弃输出层,隐藏层权重即为嵌入。诀窍是用**负采样(negative sampling)**把对 10 万词的 softmax 转成二分类。
Bir kelime, birlikte bulunduğu kelimelerden ibarettir. Bu fikri sığ bir ağ ile eğitin, geometri kendiliğinden ortaya çıkar.
Type: Build · Languages: Python · Prerequisites: Aşama 5 · 02 (BoW + TF-IDF), Aşama 3 · 03 (Sıfırdan Geri Yayılım) · Time: ~75 dakika
Problem —— TF-IDF,
dogvepuppy'nin farklı kelimeler olduğunu bilir ama neredeyse aynı anlama geldiklerini bilmez.dogüzerinde eğitilen bir sınıflandırıcı,puppyhakkındaki bir yoruma genelleme yapamaz. Eş anlamlıları listeleyerek geçiştirebilirsiniz, ama bu; nadir terimlerde, alana özgü jargonda ve öngöremediğiniz her dilde başarısız olur.
dogvepuppy'nin uzayda birbirine yakın düştüğü,king - man + woman'ınqueen'e yakın indiği bir temsil istiyorsunuz. Word2Vec bunu verdi: iki katmanlı bir ağ, trilyon token'lık eğitim, 2013'te yayımlandı.Kavram —— Dağılımsal hipotez (Firth, 1957): "Bir kelimeyi birlikte bulunduğu kelimelerden tanırsın." İki tür: Skip-gram merkez kelimeden çevredekileri tahmin eder (pencere 2 ile
cat -> (the, sat, on)); CBOW çevredekilerden merkezi tahmin eder. Ağın doğrusal olmayan aktivasyonu olmayan tek bir gizli katmanı vardır: girdi one-hot vektör, çıktı softmax'tır; eğitimden sonra çıktı katmanı atılır, gizli katman ağırlıkları gömmelerdir. Püf nokta, 100 bin kelimelik softmax'ı negatif örnekleme (negative sampling) ile ikili sınıflandırmaya çevirmektir.
Note how dog, puppy, king - man + woman, softmax, one-hot, Skip-gram, CBOW, TF-IDF, Word2Vec, and the **Type:** / **Time:** labels stay verbatim in both languages while the surrounding prose is fully localized. That preservation is enforced mechanically by the placeholder round-trip, not left to the model.