docs: measured scaling study — lane arrays, iso-chip budgets, clock cost

docs/scaling.md answers 'what does parallel instantiation buy?' with
measurements from a new probe, fpga/scale_sweep.sh (butterfly lane
arrays, each lane with its own twiddle ROM, register-wrapped like
fmax.sh with a two-stage XOR reduce so the harness never owns the
critical path).

Measured:
- Area is linear d=16..240 and K-RED's LUT premium is ~zero: a kred
  lane (butterfly + psi-fold ROM) is a few percent CHEAPER in LUTs than
  a ref lane (Barrett + full ROM) at one third the DSPs — the psi-fold
  ROM pays for the K-RED adders. The remaining trade is ~7x CARRY4.
- At equal lane count the clock gap is small: ref 74.1 vs kred
  69.2 MHz at d=32 (module-level it was -26%, core-level ~1%); kred
  degrades gracefully to 67.8 MHz at d=64.
- Operational finding: ref at d=64 = 80% DSP utilization never finished
  nextpnr placement in 15 h (dense DSP placement is the open placer's
  weak spot); the same 64 lanes are a routine 27%-DSP build with K-RED.
  kred d=128 (83% LUT) still routing; appended when done.

Analysis grounded on those numbers: iso-chip lane budgets (the retrofit
converts the binding resource from DSP to LUT; 1.0-1.5x lanes on
standalone Artix parts plus half-to-two-thirds of DSPs freed; the full
3x materializes when DSPs are contended), xc7a100t throughput shape
(~1.4x NTT/s with 120 DSPs spare), and the ceiling story (stage width
512; full 10-stage unroll = 5,120 DSPs vs 15,360 — one fits the largest
current devices, the other does not).

All caveats stated: no banked memory network in the probe, one seed,
open-flow timing, farm absolutes below isolated-module Fmax by design.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: grandchildrice

docs/README.md

@@ -12,6 +12,7 @@ claim traces to a CI-reproducible script.
 | `evaluation.md` | system-sim results + synthesis cost table (PnR TODO) |
 | `generalization.md` | Proth-prime generator; Falcon + Kyber instances |
 | `algorithm-level.md` | where the ideas apply outside hardware; prime-selection guidance (with `generator/prime_scout.py`) |
+| `scaling.md` | measured lane-array scaling: iso-chip parallelism, clock cost, the ceiling (with `fpga/scale_sweep.sh`) |
 | `paper/paper.md` | full manuscript draft (section TODOs + refs inline) |
 | `venue-assessment.md` | where-to-submit / what-level analysis |
 

docs/scaling.md

@@ -0,0 +1,137 @@
+# Scaling: what the 3→1 DSP saving buys under parallel instantiation
+
+The paper's single-BFU core is deliberately minimal. This note asks the
+next question: if you spend the saving on parallel butterfly lanes, how
+much performance does one chip buy, and where is the ceiling? Numbers
+here are measured with `fpga/scale_sweep.sh` unless marked as estimates;
+none of this is in the paper's claim set.
+
+## Method
+
+`fpga/scale_sweep.sh` synthesizes a *lane array*: d butterflies, each
+with its own twiddle ROM, fed from a shift register, outputs reduced
+through a two-stage registered XOR so the harness never owns the
+critical path (an earlier single-stage XOR tree did, and cost ~20 MHz —
+kept in the git history as a cautionary tale). Two variants:
+
+- **ref**: reference `compact_bf` (Barrett, 3 DSP) + full `tf_ROM`
+- **kred**: `compact_bf_v2` (K-RED, 1 DSP) + ψ-fold `tf_rom_fold`
+
+The probe measures how the *arithmetic* scales. It deliberately excludes
+the banked memory system, address generators and lane-to-bank
+permutation network a full d-lane core needs; those add area to both
+variants and their wiring pressure grows with d.
+
+## Measured: area scales linearly, and the LUT cost per lane is a wash
+
+yosys `synth_xilinx` (Artix-7 primitives):
+
+| variant | d | LUT | FF | DSP | CARRY4 |
+|---|---|---|---|---|---|
+| kred | 16 | 6,876 | 2,946 | 16 | 960 |
+| kred | 64 | 27,265 | 11,778 | 64 | 3,840 |
+| kred | 128 | 52,870 | 23,554 | 128 | 7,680 |
+| kred | 240 | 98,398 | 44,162 | 240 | 14,400 |
+| ref | 16 | 6,917 | 2,898 | 48 | 384 |
+| ref | 64 | 27,785 | 11,586 | 192 | 1,536 |
+| ref | 80 | 34,579 | 14,482 | 240 | 1,920 |
+
+Two facts fall out:
+
+1. **Linear**: per-lane cost is essentially constant from d=16 to d=240
+   (kred 410–430 LUT/lane, ref ~432 LUT/lane); no superlinear glue in
+   the arithmetic itself.
+2. **The LUT premium of K-RED is zero — slightly negative.** The
+   butterfly alone costs more LUTs than Barrett's (231 vs 158), but the
+   ψ-fold ROM gives back more than the difference (192 vs 241): a full
+   kred lane is a few percent *cheaper* in LUTs than a ref lane, at one
+   third the DSPs. The 3→1 DSP saving is not bought with fabric. The
+   trade that remains: kred lanes carry ~7× the CARRY4 chains
+   (shift-add adders), which is where routing pressure would appear.
+
+## Measured: post-route clock at equal lane count
+
+openXC7 nextpnr-xilinx, xc7a100t, one seed, 2.0 ns target:
+
+| variant | d | DSP util | LUT util | post-route Fmax |
+|---|---|---|---|---|
+| kred | 32 | 13% | 22% | 69.2 MHz |
+| ref | 32 | 40% | 22% | 74.1 MHz |
+| kred | 64 | 27% | 43% | 67.8 MHz |
+| kred | 128 | 53% | 83% | (routing at the time of writing; appended below when done) |
+| ref | 64 | 80% | 44% | **did not converge**: 15 h of placement without a result |
+
+Three observations:
+
+- **At equal lane count, Barrett clocks ~7% faster** (74.1 vs
+  69.2 MHz at d=32). The module-level gap was −26% in isolation; in a
+  lane array it shrinks to ~7%, and in the full single-BFU core it was
+  ~1%. So the clock price of K-RED depends on how much else surrounds
+  the butterfly, and at farm scale it is small against the 3× DSP
+  saving.
+- **kred's congestion behaviour is graceful so far**: 69.2 → 67.8 MHz
+  (−2%) from d=32 to d=64 despite the CARRY4 density.
+- **Dense DSP placement is the open flow's weak spot**: ref at d=64
+  (192 of 240 DSPs, 80%) never finished placement in 15 hours. That is
+  a statement about nextpnr-xilinx's placer, not about the Barrett
+  design, but it is operationally real if the open flow is your
+  toolchain: with K-RED the same 64 lanes are a routine 27%-DSP build.
+
+Absolute farm frequencies (~68–74 MHz) sit below the isolated-module
+Fmax (~122–164 MHz) because the harness loads every lane from one giant
+shift register with a shared select; use the *relative* numbers.
+
+## Iso-chip lane budgets
+
+With measured per-lane costs, the lane count a device supports is
+`min(DSP budget, usable-LUT budget)`; we assume 80% of LUTs are usable
+by lanes (the rest for the memory system this probe excludes):
+
+| device | LUT / DSP | ref lanes (DSP-bound) | kred lanes | lane ratio | DSPs left over (kred) |
+|---|---|---|---|---|---|
+| xc7a35t | 20.8k / 90 | 30 | ~39 (LUT-bound) | 1.3× | 51 |
+| xc7a100t | 63.4k / 240 | 80 | ~120 (LUT-bound) | 1.5× | 120 |
+| xc7a200t | 134.6k / 740 | ~246 | ~254 (LUT-bound) | ~1.0× | ~486 |
+
+The honest reading: **the retrofit converts the binding resource from
+DSP to LUT.** On a standalone Artix part the lane-count gain is
+1.0–1.5×, *and* half to two-thirds of the device's DSPs stay free. The
+full 3× materializes exactly when DSPs are the contended resource — a
+DSP-poor device, or the realistic case where the NTT shares the chip
+with other DSP-hungry logic (an FHE accelerator's multipliers, filters,
+anything). In every case the freed DSPs are the fungible currency; the
+lane ratio is what a dedicated-chip benchmark would see.
+
+Putting lanes and clocks together for xc7a100t (5,120 butterflies per
+1024-point NTT, 1 butterfly/lane/cycle; farm clocks, so shapes not
+absolutes): ref at 80 lanes and ~74 MHz gives 64 cycles ≈ 0.86 µs/NTT;
+kred at 120 lanes and ~68 MHz gives ~43 cycles ≈ 0.63 µs/NTT — about
+**1.4× the transforms per second per chip, with 120 DSPs still free**.
+
+## The ceiling
+
+Parallelism inside one transform is capped by the stage width:
+N/2 = 512 butterflies for N = 1024. Beyond that, unroll the 10 stages
+into a pipeline (one transform per cycle in steady state): 5,120
+butterflies in flight, i.e. **5,120 DSPs for K-RED vs 15,360 for
+Barrett**. The largest current devices carry ~14k DSPs, so the fully
+unrolled Falcon NTT fits on one chip with K-RED and does not with
+Barrett (estimate from device datasheets, not synthesized). That is the
+general shape of the answer: every scaling ceiling — stage width, chip
+DSP budget, board count — sits 3× further away in the resource NTT
+accelerators are usually bound by.
+
+## Reproduce
+
+```sh
+# area sweep (minutes)
+nix shell nixpkgs#yosys --command fpga/scale_sweep.sh
+# post-route Fmax points (tens of minutes each; PNR_SET picks points)
+DO_PNR=1 NP=<nextpnr-xilinx> CHIPDB=<xc7a100tcsg324.bin> fpga/scale_sweep.sh
+```
+
+Caveats: lane arrays without the banked memory network (its wiring cost
+grows with d and hits both variants); one PnR seed; open-flow timing
+model (see the paper §6's hedge); the 80%-usable-LUT assumption; ROMs
+map to distributed LUT-ROM at this size, so BRAM counts are zero in the
+probe.

fpga/scale_sweep.sh

@@ -0,0 +1,118 @@
+#!/usr/bin/env bash
+# Butterfly-farm scaling probe: how do LUT/FF/DSP scale when d butterfly
+# lanes (each with its own twiddle ROM) are instantiated side by side, for
+# the reference (Barrett, 3 DSP/lane, full tf_ROM) vs the proposed
+# (K-RED, 1 DSP/lane, psi-fold tf_rom_fold)?  Backs docs/scaling.md.
+#
+# This is a LANE-ARRAY probe, deliberately without the banked memory
+# network / AGU of a full d-lane core: it measures how the arithmetic
+# scales and where each variant hits the device's DSP/LUT wall.
+#
+# Area sweep (yosys synth_xilinx):
+#   nix shell nixpkgs#yosys --command fpga/scale_sweep.sh
+# Optional post-route Fmax points (openXC7; slow, ~10-40 min each):
+#   DO_PNR=1 NP=<nextpnr-xilinx> CHIPDB=<xc7a100tcsg324.bin> fpga/scale_sweep.sh
+set -euo pipefail
+cd "$(dirname "$0")/.."
+RTL=cfntt_ref/hardware_code_radix-2
+YOSYS=${YOSYS:-yosys}
+OUT=${OUT:-/tmp/scale_sweep}; mkdir -p "$OUT"
+
+KRED_SRC="kred-butterfly/compact_bf_v2.v kred-butterfly/modular_mul_kred.v \
+  psi-fold-rom/tf_rom_fold.v $RTL/modular_add.v $RTL/modular_substraction.v \
+  $RTL/modular_half.v $RTL/common_lib.v"
+REF_SRC="$RTL/compact_bf.v $RTL/modular_mul.v $RTL/tf_ROM.v \
+  $RTL/modular_add.v $RTL/modular_substraction.v $RTL/common_lib.v"
+
+gen_farm() { # kind d -> $OUT/farm_<kind>_<d>.v  (self-wrapped: clk/serial io)
+  local kind=$1 d=$2 bf rom
+  if [ "$kind" = kred ]; then bf=compact_bf_v2; rom=tf_rom_fold
+  else bf=compact_bf; rom=tf_ROM; fi
+  python3 - "$kind" "$d" "$bf" "$rom" > "$OUT/farm_${kind}_${d}.v" <<'PY'
+import sys
+kind, d, bf, rom = sys.argv[1], int(sys.argv[2]), sys.argv[3], sys.argv[4]
+IB = 38 * d + 1          # per lane: u14 v14 addr10; +1 shared sel
+OB = 28 * d
+print(f"// GENERATED by scale_sweep.sh: {d}-lane {kind} butterfly farm")
+print(f"module farm (input clk, input serial_in, output serial_out);")
+print(f"  reg [{IB-1}:0] ish;")
+print(f"  always @(posedge clk) ish <= {{ish[{IB-2}:0], serial_in}};")
+print(f"  wire sel = ish[{IB-1}];")
+print(f"  wire [{OB-1}:0] ow;")
+for i in range(d):
+    b = 38 * i
+    print(f"  wire [13:0] w{i};")
+    print(f"  {rom} rom{i} (.clk(clk), .A(ish[{b+37}:{b+28}]), .REN(1'b1), .Q(w{i}));")
+    print(f"  {bf} bf{i} (.clk(clk), .rst(1'b0), .sel(sel),")
+    print(f"    .u(ish[{b+27}:{b+14}]), .v(ish[{b+13}:{b}]), .w(w{i}),")
+    print(f"    .bf_upper(ow[{28*i+27}:{28*i+14}]), .bf_lower(ow[{28*i+13}:{28*i}]));")
+print(f"  reg [{OB-1}:0] osh;")
+print(f"  reg [{d-1}:0] xr1;")
+print(f"  reg xr2;")
+print(f"  always @(posedge clk) begin")
+print(f"    osh <= ow;")
+# pipeline the XOR reduce so the harness never owns the critical path:
+# per-lane 28-bit XOR, then a registered d-bit XOR, then the pin
+for i in range(d):
+    print(f"    xr1[{i}] <= ^osh[{28*i+27}:{28*i}];")
+print(f"    xr2 <= ^xr1;")
+print(f"  end")
+print(f"  assign serial_out = xr2;")
+print(f"endmodule")
+PY
+}
+
+synth_stat() { # kind d srcs...
+  local kind=$1 d=$2; shift 2
+  gen_farm "$kind" "$d"
+  local reads="" tmp="$OUT/stat_${kind}_${d}.txt"
+  for f in "$@" "$OUT/farm_${kind}_${d}.v"; do reads="$reads read_verilog $f;"; done
+  $YOSYS -p "$reads synth_xilinx -flatten -top farm; tee -o $tmp stat" >/dev/null 2>&1
+  awk -v K="$kind" -v D="$d" '
+    /^ *[0-9]+ +LUT[1-6]/ {lut+=$1}
+    /^ *[0-9]+ +FD(CE|RE|PE|SE)/ {ff+=$1}
+    /^ *[0-9]+ +DSP48E1/ {dsp+=$1}
+    /^ *[0-9]+ +RAMB(18|36)E1/ {bram+=$1}
+    /^ *[0-9]+ +CARRY4/ {carry+=$1}
+    END {printf "%-6s d=%-4d LUT=%-6d FF=%-6d DSP=%-4d BRAM=%-3d CARRY4=%-5d\n",
+         K, D, lut, ff, dsp, bram, carry}
+  ' "$tmp"
+}
+
+echo "== area sweep (yosys synth_xilinx, Artix-7 primitives) =="
+for d in 16 64 128 240; do synth_stat kred $d $KRED_SRC; done
+for d in 16 64 80;      do synth_stat ref  $d $REF_SRC;  done
+
+if [ "${DO_PNR:-0}" = 1 ]; then
+  : "${NP:?}" "${CHIPDB:?}"
+  # Which points to place-and-route, as "kind:d" pairs.  Note ref:64
+  # (192/240 DSP = 80% DSP utilization) did NOT converge in 15 h of
+  # nextpnr placement in our runs — dense DSP placement is hard for the
+  # open placer — hence ref is probed at d=32.
+  PNR_SET=${PNR_SET:-"kred:32 ref:32 kred:64 kred:128"}
+  echo "== post-route Fmax points (xc7a100t, 1 seed, target 2.0 ns) =="
+  pnr() { # kind d srcs...
+    local kind=$1 d=$2; shift 2
+    gen_farm "$kind" "$d"
+    local reads=""
+    for f in "$@" "$OUT/farm_${kind}_${d}.v"; do reads="$reads read_verilog $f;"; done
+    $YOSYS -p "$reads synth_xilinx -flatten -top farm; delete t:\$scopeinfo; \
+      write_json $OUT/farm_${kind}_${d}.json" >/dev/null 2>&1
+    { echo "set_property IOSTANDARD LVCMOS33 [get_ports {clk}]"
+      echo "set_property IOSTANDARD LVCMOS33 [get_ports {serial_in}]"
+      echo "set_property IOSTANDARD LVCMOS33 [get_ports {serial_out}]"
+      echo "create_clock -period 2.0 -name clk [get_ports clk]"; } > "$OUT/c.xdc"
+    local f log="$OUT/pnr_${kind}_${d}.log"
+    "$NP" --chipdb "$CHIPDB" --seed 1 --xdc "$OUT/c.xdc" \
+        --json "$OUT/farm_${kind}_${d}.json" > "$log" 2>&1 || true
+    f=$(grep -oE "Max frequency for clock.*: [0-9.]+ MHz" "$log" \
+        | grep -oE "[0-9.]+" | tail -1 || echo FAIL)
+    printf "%-6s d=%-4d Fmax=%s MHz  (log: %s)\n" "$kind" "$d" "${f:-FAIL}" "$log"
+  }
+  for pt in $PNR_SET; do
+    case $pt in
+      kred:*) pnr kred "${pt#kred:}" $KRED_SRC ;;
+      ref:*)  pnr ref  "${pt#ref:}"  $REF_SRC  ;;
+    esac
+  done
+fi