Add gpt-oss model (OpenAI MoE: gpt-oss-20b / 120b) to candle-transformers

[toddwbucy]

Why (WeaverTools consumer context)

WeaverTools needs gpt-oss on the native / safetensors candle path for its primary goal of model-family interchangeability: develop/train an agent on gpt-oss-20b (cheap dev tokens) and deploy that same agent on gpt-oss-120b for production. This needs the safetensors/native path — not GGUF — because the 120B target uses NVLink model-parallelism and GGUF splits across the PCI bus (the reason PR huggingface#3525 enable_peer_access exists). candle-transformers currently has no gpt-oss model.

Current state

A WIP scaffold exists on feat/gpt-oss (candle-transformers/src/models/gpt_oss.rs) — it compiles and captures the architecture + standard assembly. It is not yet load- or numerically-correct; see Definition of Done.

Architecture (reference)

• MoE, top-4 routing (20b: 32 experts; 120b: 128). Linear router with bias.
• Attention sinks — per-head learnable logit concatenated to pre-softmax scores, then dropped from the value sum (SDPA-incompatible; eager path).
• Alternating sliding-window / full attention (window 128), starting sliding.
• YaRN RoPE (theta=150000, 131k context).
• Clamped-SwiGLU experts: (up + 1) * (gate * sigmoid(1.702 * gate)), gate clamped to +limit, up to [-limit, +limit], limit = 7.0. Fused gate_up_proj + down_proj, both with bias.
• RMSNorm; attention_bias = true.
• gpt-oss-20b config: hidden 2880, intermediate 2880, 24 layers, 64 heads, 8 KV heads, head_dim 64, vocab 201088, eps 1e-5.

Definition of done

1. Fused + MXFP4 expert weights (experts.gate_up_proj[_bias], experts.down_proj[_bias], 3D) — adapt loading from the scaffold's per-expert Linears (dequantize to bf16 or handle MXFP4).
2. YaRN rope scaling wired (scaffold has a plain-RoPE stub).
3. Sliding-window mask per-layer (scaffold flags layer type but doesn't apply the window).
4. KV cache (offset-aware) for generation.
5. Numerical parity vs HF modeling_gpt_oss on gpt-oss-20b (acceptance test).
6. Loads openai/gpt-oss-20b safetensors end-to-end + generates coherent text.

Workflow

feat/gpt-oss -> fork self-PR to main (CodeRabbit pre-flight) -> upstream PR to huggingface/candle -> merge into integration -> report new integration SHA back to WeaverTools.

References

• https://huggingface.co/openai/gpt-oss-20b
• https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt_oss/modeling_gpt_oss.py
• https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt_oss/configuration_gpt_oss.py

[toddwbucy]

Fit-check complete - gpt-oss fits candle cleanly

Pulled config.json + safetensors headers for openai/gpt-oss-20b (13.76 GB, 3 shards, cached locally). Config matches the scaffold's Config::gpt_oss_20b() exactly (incl. tie_word_embeddings=false, rope_scaling=yarn factor 32 / original ctx 4096, explicit alternating layer_types).

Only the experts are MXFP4. quantization_config.modules_to_not_convert excludes self_attn, mlp.router, embed_tokens, lm_head (all bf16). So MXFP4 handling is confined to expert-weight loading in gpt_oss.rs plus a small dequant helper - no candle-core change. Guardrail check passes: this belongs upstream cleanly.

Real expert tensor layout (DoD #1)

Fused per layer (NOT per-expert Linears as the scaffold assumes):

mlp.experts.gate_up_proj_blocks  U8   [32, 5760, 90, 16]   # in 2880 = 90 blocks x 32
mlp.experts.gate_up_proj_scales  U8   [32, 5760, 90]        # e8m0 exponent per block
mlp.experts.gate_up_proj_bias    BF16 [32, 5760]
mlp.experts.down_proj_blocks     U8   [32, 2880, 90, 16]
mlp.experts.down_proj_scales     U8   [32, 2880, 90]
mlp.experts.down_proj_bias       BF16 [32, 2880]

MXFP4 dequant: each 16-byte block -> 32 nibbles (low nibble = even index, high = odd); fp4 e2m1 table [0,.5,1,1.5,2,3,4,6,-0,-.5,-1,-1.5,-2,-3,-4,-6]; value = table[code] * 2^(scale_u8 - 127). Result [experts, out, in] in [out_features, in_features] orientation -> per-expert Linear weights.

First cut: dequant to bf16 at load. Caveat: ~13 GB MXFP4 -> ~38-40 GB bf16 (native-MXFP4 inference is the later efficient path).

Gaps beyond the DoD

1. gate/up interleave: gate_up out = 5760 = 2x2880, interleaved (gate=[...,::2], up=[...,1::2]). Scaffold does a contiguous half-split - wrong. Fix rides with DoD candle-flash-attn: vendor Tri Dao FA v2.8.3 forward kernels (PR-FA-1) #1.
2. ASCII-only: scaffold uses unicode box-drawing separators and em-dashes in docstrings - must be ASCII before upstream (flagged on candle-core: add Device::enable_peer_access for cross-CUDA P2P transfers huggingface/candle#3525).
3. Sink-column narrow couples to KV-cache offset (revisit with DoD candle-core: add Device::enable_peer_access for cross-CUDA P2P transfers #4); ModelForCausalLM returns last-token logits only - parity test (DoD Add gpt-oss model (OpenAI MoE: gpt-oss-20b / 120b) to candle-transformers #5) needs all positions.

Suggested order

DoD #1 (loading + interleave) -> #3 SWA mask -> #4 KV cache -> #2 YaRN -> #5 parity gate -> #6 end-to-end.

Workflow note

This issue's "Workflow" section is superseded: we dogfood in integration first (review there), then open the Todd-gated upstream PR. feat/* still branches off main.

---

DoD #1 implemented (uncommitted on feat/gpt-oss)

• dequant_mxfp4(blocks, scales, dtype): unpacks nibbles, applies the FP4 e2m1 LUT and per-block e8m0 scale, returns [E, out, in]. CPU, once at load.
• Expert loading reworked from per-expert Linears to the fused layout: get_unchecked_dtype(.., U8) for _blocks/_scales, get for the bf16 bias, then slice per expert into Linear::from_weights.
• gate/up de-interleave fixed (reshape (.., I, 2)).
• Sink-column narrow now uses the live key count (KV-cache ready).
• Full ASCII pass on comments/docstrings.

cargo check -p candle-transformers passes (one expected dead_code warning on sliding_window -> DoD #3). Not yet validated: end-to-end load (needs ~40 GB bf16) and numerical parity.

Remaining order: #3 SWA mask -> #4 KV cache -> #2 YaRN -> #5 parity -> #6 e2e.

[toddwbucy]

DoD #1 validated end-to-end on real weights

Smoke test (CPU, f32, openai/gpt-oss-20b from cache, dummy 8-token input):

load + MXFP4 dequant : 44.8s
forward (seq 8)      : 1.47s
logits               : [1, 201088], finite=true, no NaN/Inf

So the full path is correct on real weights: MXFP4 dequant -> fused per-expert load -> gate/up de-interleave -> attention (GQA + sinks) -> MoE routing -> forward, yielding well-shaped finite logits.

The load surfaced two contiguity bugs, both strided views reaching candle custom ops; fixed by inserting .contiguous():

• attention sinks: broadcast_as(..) + a no-op to_dtype (dtypes already matched) left a stride-0 tensor feeding softmax_last_dim.
• expert gate/up de-interleave slices (.i((.., .., 0/1))) feeding clamp.

dequant_mxfp4 unit tests (nibble order / FP4 e2m1 table / e8m0 scale / output shape) pass.

NOT yet validated: numerical parity (DoD #5) - argmax on dummy tokens is meaningless until then. YaRN and the SWA mask are still unwired (no effect for seq < window=128, but required for parity).

Smoke runner is a throwaway (git-excluded), not part of the upstream PR.

Remaining: #3 SWA mask -> #4 KV cache -> #2 YaRN -> #5 parity -> #6 e2e.

[toddwbucy]

DoD #3 done: per-layer sliding-window mask

• Alternating masks now applied per layer: full-causal for full-attention layers, sliding-window (window 128) for the sliding layers; selected in Model::forward via a per-layer is_sliding flag.
• Refactor: removed the unused sliding_window field from Attention (it just applies the mask it is handed). This resolves the dead_code warning -> cargo check -p candle-transformers is now clean (ready for -Dwarnings).
• causal_mask(seq_len, offset, window) is offset-aware (forward-compatible with the DoD candle-core: add Device::enable_peer_access for cross-CUDA P2P transfers #4 KV cache). Sliding rule: masked when q_abs - kj >= window (HF-strict). NOTE: candle's gemma2 uses > window (off-by-one) - confirm the exact gpt-oss convention at parity (Add gpt-oss model (OpenAI MoE: gpt-oss-20b / 120b) to candle-transformers #5).
• Smoke test extended to seq 160 (> window) to actually exercise the clip: loads + forward (4.2s), finite logits. dequant unit tests still green.

Remaining: #4 KV cache -> #2 YaRN -> #5 parity -> #6 e2e.

[toddwbucy]

DoD #4 + #2 done (committed on feat/gpt-oss)

#4 KV cache (0834efe8): candle_nn::KvCache wired into attention; forward is now &mut self; clear_kv_cache exposed. Masks are built every step so sliding layers mask beyond-window keys during single-token decode. Smoke: prefill (seq 160) 4.3s, decode ~0.45s/tok, finite logits throughout. Plain KvCache + sliding mask (RotatingKvCache is a later memory optimization).

#2 YaRN (e6ca59ae): inverse-frequency interpolation per rope_scaling (extrapolate high-freq dims, interpolate low-freq by 1/factor with a beta_fast/beta_slow ramp) + mscale folded into cos/sin. Unit-tested. truncate flag not yet handled.

Commits: 15003028 (#1 + #3), 0834efe8 (#4), e6ca59ae (#2).

Remaining: #5 numerical parity vs HF (the correctness gate - pins the sliding-window off-by-one, YaRN truncate, sink exactness, gate/up interleave) -> #6 end-to-end generate.

[toddwbucy]

DoD #5 closed: numerical parity vs HF confirmed

Reference: HF GptOssForCausalLM (bf16, Mxfp4Config(dequantize=True), GPU) vs candle (f32, CPU), identical input_ids, comparing last-position logits.

prompt	seq	argmax (ours/ref)	cosine	top5
"The capital of France is"	5	12650 / 12650 (match)	0.99981	4/5
long passage (>window)	341	623 / 623 (match)	0.99949	4/5

The 341-token run exercises the sliding-window clip (sliding layers attend only to the last 128 keys); a matching argmax there confirms the window boundary (HF-strict q - k >= window) is correct. Both runs match argmax with cosine ~0.999; the residual (max-abs ~0.5-0.8, the differing 5th-ranked token) is bf16-vs-f32 precision noise, not algorithmic. This validates the gate/up interleave, attention sinks, and YaRN end to end.

(Parity harness - HF dump script + candle compare-runner - is throwaway and git-excluded.)

DoD #1-#5 are complete and committed on feat/gpt-oss. Remaining: #6 end-to-end generate (greedy/sampled decode + tokenizer), then the integration-first review flow toward an upstream PR.

[toddwbucy]

gpt-oss-20b is functional and verified - ready to dogfood on integration

All of the Definition of Done is implemented on feat/gpt-oss and merged into integration:

• candle-flash-attn: vendor Tri Dao FA v2.8.3 forward kernels (PR-FA-1) #1 MXFP4 fused expert loading - localized dequant to bf16 at load, no candle-core changes
• candle-flash-attn: extend FFI for split-KV dispatch (PR-FA-2) #2 YaRN RoPE, candle-flash-attn: wire split-KV heuristic + accumulator alloc (PR-FA-3) #3 per-layer sliding-window attention, candle-core: add Device::enable_peer_access for cross-CUDA P2P transfers #4 offset-aware KV cache
• Add gpt-oss model (OpenAI MoE: gpt-oss-20b / 120b) to candle-transformers #5 numerical parity vs HF modeling_gpt_oss: argmax match + cosine 0.999 on both a short prompt and a 341-token prompt (the latter exercises the sliding-window clip)
• Add the gpt-oss model (OpenAI MoE: gpt-oss-20b / gpt-oss-120b) #6 end-to-end example candle-examples/examples/gpt-oss: generates coherent text ("The capital of France is Paris.")

Pin this integration SHA: a9fe2926 (current integration + the four gpt-oss commits; cargo check --workspace is clean).

Please pin a9fe2926 and run gpt-oss-20b through the same path you use for qwen and gemma (safetensors / native, model-parallel). Run on GPU with bf16 (~40 GB fits a single 48 GB card; CPU works but is slow). If it holds up there, gpt-oss is the third model family supported and we proceed to the upstream PR.

Upstream landscape (FYI, for when we go up): there is a 3-month-dormant PR huggingface#3391 that takes the heavy path (native MXFP4 in candle-core + CUDA/Metal kernels, 0 reviews) and a closed non-functional huggingface#3129. Ours is the lighter, parity-verified alternative - we'll frame the PR to credit both once you confirm.

[toddwbucy]

Overnight DoD validation + parity-residual investigation — results

Ran the full DoD validation on feat/gpt-oss @ ca8898b9 (RTX A6000, CUDA 13.0, transformers 4.57.6). All three DoD acceptance checks pass, and the small parity residual is fully explained as bf16 quantization floor — not an implementation bug.

DoD checks

DoD	Check	Result
#1	smoke — load gpt-oss-20b, dequantize MXFP4, one forward, finite logits (CPU/F32)	✅ load+dequant 44.8 s; prefill seq 160 (past the 128 sliding window so the clip engages); offset-aware KV-cache decode steps clean; SMOKE OK
#5	parity — candle logits vs HF modeling_gpt_oss	✅ argmax match, top-4 exact, cosine 0.9996 (see below)
#6	generation — coherent text (GPU0/CUDA bf16)	✅ "The capital of France is Paris." + coherent continuation, 22 tok/s

Parity residual — root cause

The first parity run (candle CPU/F32 vs the dumped reference) gave cosine 0.999486, max_abs 0.8282, top5 4/5. Chasing that residual:

1. The reference was computed in bf16, candle in F32 — not apples-to-apples. gptoss_ref.py loads with dtype=torch.bfloat16 and casts to F32 only when saving the logits. So the comparison was bf16-vs-f32, whose signature (high cosine, argmax+top-4 exact, a single tail-token swap, ~0.8 max-abs) is textbook bf16 reference noise.

2. A pure-F32 reference is not obtainable from HF. Mxfp4Config(dequantize=True) dequantizes the experts to bf16 regardless of the model dtype — requesting dtype=torch.float32 crashes in the expert matmul (expected m1 and m2 to have the same dtype, but got: float != c10::BFloat16). HF's experts are inherently bf16, so bf16-vs-bf16 is the tightest apples-to-apples comparison possible.

3. Matched bf16-vs-bf16 (candle GPU0/bf16 vs the bf16 reference):

cosine = 0.999560   max_abs_diff = 0.7344   top5_overlap = 4/5
absdiff: median 0.1250  p99 0.4062  max 0.7344  | >0.1: 121619  >0.3: 13202  >0.5: 195  of 201088

Median abs-diff is 0.125 = 1–2 bf16 ULPs at logit magnitude ~10.5 (bf16 step there is 0.0625). That is the floor of bf16 representability — two independent bf16 computations with different reduction orders cannot agree more tightly than ~1 ULP. Only 195/201088 logits (0.1%) exceed 0.5.

4. The "top5 4/5" swap is a genuine near-tie, and candle is the more-precise side. Per-token logits at the swapped tail:

token	candle bf16	HF bf16	candle F32
623 (argmax)	12.0000	11.9375	12.0175
16	11.3125	11.3125	11.2757
2550	11.0000	10.9375	10.8978
17	10.6875	10.7500	10.6249
35373	10.5625	10.4375	10.5603
18	10.4375	10.4375	10.3302

HF's bf16 collapsed 35373 and 18 to the identical value 10.4375; the index tiebreak put 18 in the top-5. In F32, candle cleanly separates them (35373 10.560 > 18 10.330, by 0.23) — i.e. candle's more-precise computation says 35373 genuinely outranks 18. The swap is HF losing the distinction to bf16 rounding, not candle mis-ranking.

Conclusion

The gpt-oss-20b port is numerically faithful — DoD #5 is met. The residual is bf16 quantization at the representability floor (median ≈ 1 ULP), the single top-5 swap is a bf16 near-tie that candle resolves more correctly in F32, and argmax + top-4 are exact in both dtypes. No model-side bug (MXFP4 dequant / attention-sink / YaRN / sliding-window) is implicated.

Reproduce

The throwaway parity harness was made device/dtype-configurable:

# bf16 on GPU0 vs the bf16 HF reference (the apples-to-apples check)
HF_HOME=/opt/weaver/huggingface HF_HUB_OFFLINE=1 \
CUDA_VISIBLE_DEVICES=0 PARITY_DEVICE=cuda PARITY_DTYPE=bf16 \
  cargo run --release --features cuda --example gpt-oss-parity
# (defaults to CPU/F32; PARITY_REF=<path> overrides the reference)

Remaining before upstream

• DoD candle-flash-attn: vendor Tri Dao FA v2.8.3 forward kernels (PR-FA-1) #1/Add gpt-oss model (OpenAI MoE: gpt-oss-20b / 120b) to candle-transformers #5/Add the gpt-oss model (OpenAI MoE: gpt-oss-20b / gpt-oss-120b) #6 satisfied. Next per the workflow: feat/gpt-oss → fork self-PR (CodeRabbit) → upstream PR to huggingface/candle.
• Optional nicety: a chat template / stop handling for the generation example (currently base-model continuation rambles after the correct answer).

[toddwbucy]

Request: full real-world harness validation (before any upstream resubmission)

The earlier overnight run validated gpt-oss at the component level - load/dequant smoke, HF logit parity, single-GPU generation. That is not a demonstration that gpt-oss runs in our harness as designed, end to end. We're treating that gap as the reason not to land upstream yet (the upstream PR has been withdrawn; the tracking issue stays open).

Please run gpt-oss through the real deployment path and report.

What to pin

integration @ 72cf20a0 - this now carries the cleaned-up gpt-oss (single squashed commit, review fixes + fmt applied) alongside the existing features (qwen2-rope, flash-attn, peer-access). NOTE: the previous integration a9fe2926 had a stale gpt-oss (pre-review); do not test that one.

What to validate

• gpt-oss-20b run end to end through the harness as designed (the same deployment/agent path you use for qwen and gemma): load, inference, the full loop.
• Report: does it run cleanly? throughput, memory, and any integration friction (API surface, tokenizer/stop handling, KV cache behavior under your loop, etc.).

Known limitation - please confirm against "as designed"

The candle gpt-oss model is currently single-device: VarBuilder on one GPU, no model-parallel sharding. The NVLink peer-access primitive (huggingface#3525 upstream) exists but is not wired into gpt-oss. So:

• gpt-oss-20b on a single GPU (fits one 48GB A6000): should work - this is the test above.
• gpt-oss-120b / model-parallel over NVLink (the actual model-family-interchangeability target): not possible with the current model - it needs sharding logic we have not built.

So the key question for "as designed": does the harness require model-parallelism (now, or for the 120b deploy target)? If yes, that becomes the next candle work item (converging the gpt-oss model with the peer-access path) and is the real blocker before upstream - more important than the PR itself.

Upstream resubmission (gpt-oss and the qwen2 mask fix) is on hold until this real-world validation passes.

[toddwbucy]

WeaverTools harness-side findings — integration is not a drop-in; two paths

Picking this up from the consumer (harness) side to scope the real-world validation. Pinned integration @ 72cf20a0, weights openai/gpt-oss-20b cached, hardware 2x A6000 (48GB) + 1x RTX 2000 Ada (16GB). One correction to the framing in the request above, then the integration options.

Correction: weaver's decoder is not a candle-transformers runner

The request says "run gpt-oss through the same native path you use for qwen and gemma (safetensors / native)." That conflates two different execution paths in WeaverTools:

• Encoder (Jina V4) does use candle-transformers — weaver-spu/src/models/qwen2_5_vl.rs and encoder/qwen25vl_loraed.rs wrap candle_transformers::models::qwen2::Model via VarBuilder. This is the path that consumes candle model code directly.
• Decoder (text generation) is weaver's own hand-rolled CUDA engine — weaver-spu/src/core/gpu/forward.rs is cudarc + CudaBlas/Gemm + custom kernels, loading safetensors through weaver's own decoder/model/loader.rs -> core::gpu::sharding::shard_model_to_gpu -> GpuModelWeights, with per-architecture variants (Standard vs Gemma4). Qwen and Gemma decode through this, not through candle-transformers.

Consequence: the validated candle-transformers/src/models/gpt_oss.rs cannot just be pointed at by the decoder path. qwen/gemma generation in weaver are weaver's own reimplementations. So "same path as qwen/gemma" is not a no-op — it's a real integration, and there are two ways to do it.

The seam that makes reuse possible

The decoder runs backends behind a small trait, InferenceBackend (decoder/llama_cpp_backend.rs): model_id, generate, vocab_size, cache_usage, reset_cache, model_info (+ two defaulted), held in a BackendRegistry of Arc<dyn InferenceBackend>. Two impls exist today: NativeBackend (weaver's CUDA engine) and the llama.cpp/GGUF backend. This trait is the clean reuse point.

Path A — wrap candle's model behind a 3rd InferenceBackend impl

A new backend that loads gpt_oss::Model via VarBuilder (exactly as the encoder already loads candle models) and implements generate/reset_cache/etc. candle's model already carries the offset-aware KV cache + generation, so this reuses 100% of the parity-validated numerics (MXFP4 dequant, sinks, SWA, YaRN).

• Pros: least new surface; reuses validated code; 20b bf16 (~40GB) on one A6000; this is exactly the deliverable this issue is waiting on before upstream.
• Cons: single-device only — candle's model has no model-parallel sharding, so this path does not reach gpt-oss-120b / NVLink. Adds a second decoder execution path to maintain.

Path B — port gpt-oss into weaver's own core::gpu engine

A forward_gptoss.rs + GpuModelWeightsGptOss + MXFP4 dequant in weaver's loader, mirroring the existing forward_gemma4.rs precedent, with candle's implementation as the line-by-line oracle.

• Pros: inherits weaver's TP sharding (sharding.rs), which is the only route to the 120b-over-NVLink model-family-interchangeability goal (develop on 20b, deploy the same agent on 120b).
• Cons: much larger CUDA-kernel effort; re-derives attention-sinks / alternating-SWA / YaRN / clamped-SwiGLU-MoE that candle already validated.

Answer to this issue's model-parallelism qu

does the harness require model-parallelism (nrget)?

• 20b: No. Fits a single 48GB A6000 in bf16; single-device is sufficient.
• 120b (the interchangeability target): Yesal point: weaver's TP lives in its ownengine (core::gpu::sharding), while candle's gpt-oss model is single-VarBuilder. So the 120b path is reachable either by candle wiring candle-core: add Device::enable_peer_access for cross-CUDA P2P transfers huggingface/candle#3525 peer-access into gpt_oss.rs, or by Path B (weaver's engine already shards). That
  convergence — not the upstream PR — is the real flagged.

Proposed plan

• This round (pre-upstream): Path A to validate gpt-oss-20b end-to-end through the harness as designed — a HeroBench
  agent through the daemon (load -> turns -> toke, plus a throughput / memory /integration-friction report (tokenizer/stop handling, KV-cache behavior under the agent loop, API surface). This unblocks
  the upstream resubmission.
• Follow-on (120b deploy target): Path B (or candle candle-core: add Device::enable_peer_access for cross-CUDA P2P transfers huggingface/candle#3525-into-gpt_oss) as the deliberate next work item.

Open decisions on the harness side: (1) Path A now vs A-now-then-B vs straight-to-B; (2) whether the upstream gate is "20b
end-to-end" or must also demonstrate 120b models here once the harness session lands the chosen
path.
Also worth noting for the upstream PR: harness prompt format**(<|start|><|channel|><|message|><|end|>, roles + analysis/commentary/final channels) and the o200k_harmony tiktoken tokenizer wiring — neither is required by candleration example's stop handling is the same gap(you noted the base-model continuation rambles). Aligning the example's stop tokens with Harmony would help both sides.
Two quick flags before you go: - .local-deny-patterns.txt is still not ignoredclude) so it can't get committed by accident.Say the word and I'll do just that.

• History rewrite is parked, not done — wheneve, the plan and caveats are in my last message.

[toddwbucy]

Decision (recommendation for the WeaverTools port): take Path B

Following the harness-side findings above, the call is Path B - port gpt-oss into weaver's own core::gpu engine (a forward_gptoss.rs + GpuModelWeightsGptOss + MXFP4 dequant in weaver's loader, mirroring the forward_gemma4.rs precedent), not Path A (wrapping candle's model as a 3rd InferenceBackend).

Rationale

• The 120b / NVLink-TP work has to be done in weaver's engine regardless - that is the model-family-interchangeability target. Path A would ship a second, single-device decode path that we'd have to maintain and still replace later: a foot-gun and a deferred-work trap.
• Doing Path B once, now, while candle's parity-validated gpt_oss.rs is fresh as the oracle, is cleaner.
• No urgency to land upstream; resubmission waits until the port proves it locally.

candle's role under Path B

• candle-transformers/src/models/gpt_oss.rs (integration 72cf20a0, = feat/gpt-oss 3eb54c3b) is the line-by-line numerical oracle for the port - not a runtime dependency.
• The candle side can produce per-op reference tensors on demand for kernel-by-kernel diffing: MXFP4-dequantized expert weights, attention output with sinks, alternating-SWA mask effect (seq > 128), YaRN cos/sin tables, clamped-SwiGLU MoE expert output + router top-k.

Plan / gate

• Build Path B in weaver; validate gpt-oss-20b end-to-end (and potentially 120b) locally.
• The final decision (upstream resubmission, etc.) is made on the WeaverTools side once that is proven.
• Upstream (gpt-oss + the qwen2 Qwen2: prepare_attention_mask is bidirectional (no causal triangle) when attn_mask provided huggingface/candle#3567 fix) is paused until then.

candle session is standing down until the local Path B validation is in.

[toddwbucy]

A6 closed: candle's MXFP4 dequant verified bit-exact vs HF (candle = trusted oracle)

Ran the per-op check the audit flagged as the open gap: candle's dequant_mxfp4 vs HF transformers' convert_moe_packed_tensors on the same raw blocks/scales from real gpt-oss-20b (layer 0, expert 0, both projections):

gate_up_proj: max_abs_diff vs HF = 0e0
down_proj:    max_abs_diff vs HF = 0e0

Exact (0.0), not within-tolerance. So the co-validation hazard is closed at the dequant level - candle's MXFP4 unpack matches HF ground truth, and any MX3 dequant mismatch in the Path-B kernel is a weaver-GPU bug (bucket c), not a mirrored candle bug. (Verified via a throwaway in-module test; not committed to gpt_oss.rs.)

Critical finding the per-op check surfaced (end-to-end cosine could not)

HF and candle dequant into transposed orientations:

• HF convert_moe_packed_tensors -> [E, in, out] (gate_up [E, 2880, 5760]), matching gpt-oss's stored param + its x @ W.
• candle dequant_mxfp4 -> [E, out, in] ([E, 5760, 2880]), fed to a Linear doing x @ W^T.

Same math, transposed storage; values identical under transpose. The Path-B kernel must pick an orientation explicitly and align its diffs (transpose vs candle, or match HF's [E, in, out]).

Audit summary (full detail handed to the Path-B session)

• A1 dequant formula (LUT / 2^(scale-127) / nibble low->even,high->odd): confirmed, matches HF.
• A2 in_features = nb*32: confirmed. A5 down_proj straight (no interleave): confirmed.
• A3 gate/up de-interleave on the 2I (out) axis = dim1 of raw blocks: confirmed equivalent to candle's split.
• A4: candle dequants -> bf16 at load + bf16 GEMM (f32 accum). For the f16 native path: dequanted weights are value-exact (demand 0 diff), but GEMM/logits are f16-vs-bf16 -> ~bf16 tolerance, NOT bit-exact (won't reproduce the exact 0.9997). f16 exponent range (+-15) is narrower than bf16 - assert/handle out-of-range scales.

Net: candle is a verified per-op oracle for MXFP4; treat dequant diffs as exact-required and forward/logit diffs at bf16 tolerance.