browser-lora-onboarding

Root-causing LoRA fine-tuning failures and building a repeatable method for onboarding new LLM architectures to a browser-side training engine.

A standalone research project with two goals: (1) explain why LoRA fine-tuning silently produces wrong models on certain architectures, and (2) turn each answer into reusable code that lets a browser LoRA engine train a new model family it had never seen. It has zero source dependency on any other project — see vendor/ for the small, frozen slice of transformer forward/backward code the experiments run against.

Every architecture claim here is gated by a numeric forward-parity check against the reference transformers implementation on real weights — not "loss went down," but "the logits match to bf16/f32 tolerance over the full vocabulary."

---

Contents

• Results at a glance
• Quickstart
• The onboarding toolkit
• Case studies
• Forward-parity: the credibility gate
• Repository layout
• Reproduce everything
• Method & principles
• Scope & honest limits
• Citation
• License

---

Results at a glance

Three architecture families, three answers — each verified on the real checkpoint:

Family	Mixer type	The question	Outcome
Granite 4.0	dense + muP scalars	why do trained models "collapse to one token"?	Undertraining, not a bug. muP scalars attenuate the LoRA gradient; a healthy LR trains cleanly. Confirmed on real granite-4.0-350m.
LFM2.5	conv / attention hybrid	can the engine handle a per-layer conv/attention hybrid at all?	Yes. Full hybrid runtime built; trains end-to-end; forward-parity passed (cosine 0.99999999987).
Qwen3.5-0.8B	Gated DeltaNet (linear attn)	can the engine learn a linear-attention family it had never seen?	Yes. New Gated-DeltaNet mixer ported from the reference source; trains end-to-end; forward-parity passed (cosine 0.9999999999992).

The classifier that decides "which of these is this model, and what does it need?" is generalized into the toolkit/ so the same method applies to any Hugging Face model.

---

Quickstart

Requirements: Node ≥ 22.6 (the project runs TypeScript directly via Node's type stripping; no build step). A GPU/browser is not needed — every experiment runs on the TensorFlow.js CPU backend under Vitest.

npm install
npm test          # all case studies + toolkit (CPU backend, Vitest) — 29 tests
npm run typecheck # strict TypeScript, no emit

Classify any model straight from the Hugging Face Hub (config + weight header only, no multi-GB download):

npm run probe -- ibm-granite/granite-4.0-350m   # → muP-scalars (dense, not the MoE its class name implies)
npm run probe -- LiquidAI/LFM2.5-1.2B-Instruct  # → hybrid-dispatch (10 conv + 6 attention)
npm run probe -- Qwen/Qwen3.5-0.8B              # → gated-deltanet-dispatch (18 linear_attn + 6 attention)
npm run probe -- Qwen/Qwen3-0.6B                # → as-is (plain dense transformer)

---

The onboarding toolkit

toolkit/ turns the case studies into a scripted, three-step process for taking a new HF model to clean LoRA fine-tuning:

# 1. Classify — what is this model, and what does it need?
npm run probe -- <hf-repo>

# 2. Recipe — what LoRA learning rate trains it without collapse?
npm run recipe-sweep -- --repo <hf-repo>

# 3. Parity — verify logits match transformers before shipping (toolkit/parity.ts)

The classifier reads layer_types, experts, muP scalars and the weight header — never the class name — so it correctly reports that a GraniteMoeHybridForCausalLM dense checkpoint is a muP transformer (not a hybrid or MoE), unwraps nested multimodal configs, and flags selective-SSM parameters (A_log/dt_bias) hiding under an innocuous linear_attention layer type. Verdicts range from as-is (trains today) through muP-scalars / hybrid-dispatch / gated-deltanet-dispatch (supported paths) to honest mamba-unsupported / moe-unsupported hard stops.

---

Case studies

Granite 4.0 — muP × LoRA · granite-mup/

The "trained Granite answers every prompt with a single ." bug is undertraining of a muP-parametrized model under LoRA, not an architecture or generation defect. Granite's four muP scalars (embedding ×12, attention 1/head_dim, residual ×0.263, logits ÷4) sit between the LoRA adapter and the loss and attenuate its gradient. A CPU grid maps a strictly monotonic LR boundary (all seeds collapse when undertrained, all train at a healthy LR); the intuitive "divide LR by residual_multiplier" fix moves the wrong way. Confirmed on real granite-4.0-350m (WebGPU, LoRA r=8): loss 4.66 → 0.00003 over 40 steps, correct generation from step ~16, zero collapse.

LFM2.5 — conv/attention hybrid · lfm2-hybrid/

LFM2.5 interleaves 10 gated short-conv layers with 6 GQA-attention layers. The engine assumed attention on every layer; this adds the per-layer conv/attention dispatch, a SwiGLU FFN, and a conv-state generation cache whose incremental decode provably matches full recompute. Real-weights forward-parity passed on LFM2.5-1.2B-Instruct (bf16, 2.34 GB): maxAbsErr 1.8e-5, cosine 0.99999999987, argmax and top-5 agree over the full 65 536-way vocab.

Qwen3.5-0.8B — Gated DeltaNet · qwen3.5-linear-attn/

Qwen/Qwen3.5-0.8B is a vision-language model whose text backbone interleaves 18 linear_attention layers (a Gated DeltaNet recurrence — the A_log / dt_bias signature) with 6 full_attention layers. Two toolkit blind spots surfaced and were fixed first (nested text_config, linear_attention hiding real recurrence params), then the mixer was ported line-for-line from the reference transformers.models.qwen3_5 source: causal depthwise conv, L2-normed q/k, a sequential delta-rule recurrence with a learned decay gate, gated-QKV attention, and this model's (1 + weight) RMSNorm variant. Real-weights forward-parity passed (first attempt): maxAbsErr 1.5e-5, cosine 0.9999999999992, argmax and top-5 agree over the full 248 320-way vocab. The backward pass comes free from TensorFlow.js autodiff — no manual gradient code.

---

Forward-parity: the credibility gate

"Loss went down" is not "the model is correct" — the Granite investigation began precisely because a near-zero loss coexisted with garbage output. Two checks turn a plausible port into a verified one (toolkit/parity.ts):

1. Self-parity — a model's incremental cached decode must emit the identical token sequence to a full recompute. Catches KV-cache / conv-state / recurrent-state bugs, the #1 silent-wrongness source in a hybrid.
2. Logits-parity — the engine's next-token logits must match HF transformers within bf16/f32 tolerance on a real prompt. Catches wrong split order, misplaced norm, wrong scalar — the things a random-weight stand-in cannot.

Model	vocab	maxAbsErr	cosine	argmax	top-5
LFM2.5-1.2B	65 536	1.8 × 10⁻⁵	0.99999999987	✅	✅
Qwen3.5-0.8B	248 320	1.5 × 10⁻⁵	0.9999999999992	✅	✅

---

Repository layout

browser-lora-onboarding/
├── granite-mup/          # Case study 1: muP × LoRA undertraining
│   ├── grid.test.ts        # LR × seed × difficulty collapse boundary (CPU stand-in)
│   └── RESULTS.md
├── lfm2-hybrid/          # Case study 2: conv/attention hybrid runtime
│   ├── lfm2-model.ts       # per-layer conv|attention dispatch + conv-state cache
│   ├── lfm2-loader.ts      # real safetensors → model mapping
│   ├── verify-parity.ts    # real-weights forward-parity harness
│   ├── *.test.ts           # forward / grad / train / generate / cache-parity
│   └── RESULTS.md
├── qwen3.5-linear-attn/  # Case study 3: Gated DeltaNet (linear attention)
│   ├── qwen35-model.ts     # linear_attn|attention dispatch + recurrent-state cache
│   ├── qwen35-loader.ts    # real safetensors → model mapping
│   ├── verify-parity.ts    # real-weights forward-parity harness
│   ├── *.test.ts
│   └── RESULTS.md
├── toolkit/              # Generalized onboarding method (probe → recipe → parity)
│   ├── classifier.ts       # config → architecture verdict (pure, dependency-free)
│   ├── hub.ts              # HF config + safetensors-header fetch (incl. sharded)
│   ├── safetensors-reader.ts  # local bf16/f32 safetensors reader (Node)
│   ├── probe.ts            # CLI: classify a live repo
│   ├── recipe-sweep.ts     # CLI: map the LR/collapse boundary
│   ├── mup-standin.ts      # compact muP transformer for the sweep
│   ├── parity.ts           # self-parity + logits-parity harness + reference protocol
│   └── README.md
└── vendor/engine/        # Frozen snapshots of transformer fwd/bwd + LFM2 conv mixer

Each vendored file's header documents exactly what it is a snapshot of and why the copy exists; vendored code does not auto-track upstream.

---

Reproduce everything

The CPU experiments (no download, no GPU):

npm install
npm test

The real-weights forward-parity checks additionally need Python with torch + transformers (the independent oracle) and the model checkpoints. Reference logits are committed (*/reference.json) so the comparison is reproducible even without regenerating them:

# 1. (optional) regenerate the transformers reference for a prompt
python lfm2-hybrid/gen_reference.py LiquidAI/LFM2.5-1.2B-Instruct \
  "What is the capital of France?" > lfm2-hybrid/reference.json

# 2. download the checkpoint (config.json + model.safetensors) into <ckpt-dir>, then:
npm run verify:lfm2   -- <ckpt-dir> lfm2-hybrid/reference.json
npm run verify:qwen35 -- <ckpt-dir> qwen3.5-linear-attn/reference.json

Checkpoints (*.safetensors) are git-ignored — never commit multi-GB weights.

---

Method & principles

• CPU stand-ins pin mechanisms; real weights confirm. A 2-layer random model sits in a different basin than a 28-layer pretrained one, so absolute hyperparameters don't transfer — but failure shapes do, and they are cheap and deterministic to map. Every mechanism claim is then checked on a real checkpoint where feasible.
• Generation is tested, not assumed. Every experiment ends in a greedy-decode check; "loss went down" is never the finish line.
• The name lies — read the weights. Granite's class name says MoE hybrid (it's dense muP); Qwen3.5's linear_attention layer type hides a real recurrence. The toolkit classifies on layer_types + weight header, never the advertised class name.
• No silent caps. Where a run is n=1 or a stand-in, the docs say so.

---

Scope & honest limits

• Supported today: dense attention (as-is), dense muP (Granite), conv/attention hybrids (LFM2), and Qwen3.5-style Gated-DeltaNet/attention hybrids — including scaling within each family across model sizes (dims are read from the config, not hard-coded).
• Verified, not exhaustive: each forward-parity result is currently n=1 prompt on one checkpoint size; extending to more prompts/seeds/sizes is cheap and is the remaining step before declaring a family "shippable" broadly.
• A different linear-attention model (Qwen3-Next, RWKV, others) is not auto-covered by the Qwen3.5 port — the recurrence, gating, or norm formula may differ, so the classifier reports linear-attn-unsupported rather than assuming reuse.
• Genuinely out of scope: real Mamba/SSM (mamba-unsupported) and MoE routing (moe-unsupported) — those kernels do not exist in this engine yet.

---

Citation

If you use this work, please cite it — see CITATION.cff, or:

@software{browser_lora_onboarding_2026,
  title  = {browser-lora-onboarding: teaching a browser LoRA engine to train new LLM architectures},
  year   = {2026},
  note   = {Apache-2.0},
  url    = {https://github.com/OWNER/browser-lora-onboarding}
}

License

Apache License 2.0. Vendored code in vendor/ carries its provenance in each file header.