Skip to content

Commit b9b7bef

Browse files
zh4ngxclaude
zh4ngx
and
claude
authored
docs: Task 11 — DeepSeek Engram offload ByteDMD verification (#85)
Promote issue #77 from a GitHub-only observation into a durable repo task following the pattern from PR #73 (task spec + agent prompt + findings doc path). Keeps the plan discoverable to non-GitHub-indexed agents (Gemini, Qwen, Kimi) and survives issue lifecycle. - docs/tasks/011-engram-offload-bytedmd.md — task spec, status tracker, decision thresholds (2-5x WIN, >20x LOSS-for-claim, 5-20x ambiguous) - docs/agent-prompts/engram-offload.md — reusable two-phase prompt: Phase 1 paper extraction (no run without m:M ratio and heuristic), Phase 2 microbenchmark (resident vs naive-offload vs prefetch variants) - docs/tasks/INDEX.md — row added Resolves #77 once merged (close with link to task doc). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent fbb96f5 commit b9b7bef

3 files changed

Lines changed: 212 additions & 0 deletions

File tree

‎docs/agent-prompts/engram-offload.md‎

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
# DeepSeek Engram Offload — ByteDMD Verification
2+
3+
You are contributing to the **Sutro Group**, a research lab studying energy-efficient AI training. Your task is to verify whether the DeepSeek Engram paper's "<3% offload overhead" claim survives when priced under ByteDMD (byte-level data movement), rather than wall-time.
4+
5+
This is a **two-phase task**. Phase 1 is a paper-reading pass; Phase 2 is a microbenchmark. **Do Phase 1 first** — without the paper's actual `m:M` ratio and prefetch heuristic, Phase 2 models a strawman.
6+
7+
## Project context
8+
9+
Read these files first:
10+
11+
- `DISCOVERIES.md` — what's already known
12+
- `LAB.md` — lab rules (important: do NOT modify measurement code: `tracker.py`, `cache_tracker.py`, `data.py`, `config.py`, `harness.py`, `fast.py`, `src/bytedmd/`)
13+
- `docs/research/bytedmd.md` — ByteDMD metric spec
14+
- `docs/tasks/011-engram-offload-bytedmd.md` — task spec and decision thresholds
15+
- `findings/_template.md` — expected report format
16+
17+
Key context:
18+
19+
- Our metric: **ByteDMD** — byte-granularity data movement. Reading a value at stack depth `d` costs `ceil(sqrt(d))` per byte. Writes are free. Reference: https://github.com/cybertronai/ByteDMD
20+
- The central suspicion: offloading to SSD means reads at stack depth ~`1e6`, which should cost `sqrt(1e6) = 1000` per byte — roughly 31× the cost of HBM reads at depth ~`1e3`. A 3% wall-time overhead claim is hard to reconcile with that unless the heuristic genuinely reduces *how many* deep reads happen (not just hides their latency via async prefetch).
21+
22+
## Phase 1 — Paper extraction
23+
24+
### Read the paper carefully
25+
26+
Primary source: https://deepseek.ai/blog/deepseek-engram-v4-architecture
27+
28+
Extract the following with quoted evidence from the paper where possible:
29+
30+
1. **Overhead type** — does the paper report wall-time, FLOPs, energy, or tokens/sec? The "3%" figure applies to which metric specifically?
31+
2. **`m:M` ratio** — resident set size (in cache/HBM) vs offloaded set size. Can be tokens, KV entries, parameters — whatever the paper offloads. If not stated explicitly, infer from reported configurations.
32+
3. **Lookup frequency `p`** — fraction of forward-pass lookups that hit the offloaded set.
33+
4. **Prefetch heuristic** — random? LRU? model-guided prediction of hot entries? Learned routing? Copy-on-access? Describe the mechanism in enough detail to implement a reduced version.
34+
5. **What "offload" physically means** — CPU RAM? NVMe SSD? Remote tier? All of the above?
35+
36+
### Write `docs/findings/engram-offload-paper-read.md`
37+
38+
Use this structure:
39+
40+
```markdown
41+
# DeepSeek Engram — Paper Extraction
42+
43+
## Source
44+
[URL, date accessed, paper version]
45+
46+
## Overhead claim
47+
[Exact quote. What metric does "3%" refer to?]
48+
49+
## Offload configuration
50+
- m (resident): [size, units, quoted source]
51+
- M (offloaded): [size, units, quoted source]
52+
- m:M ratio: [derived]
53+
- p (lookup fraction): [quoted or inferred]
54+
- Physical tier: [CPU RAM / SSD / other]
55+
56+
## Prefetch heuristic
57+
[2-4 paragraphs describing the mechanism, with code-level detail if possible]
58+
59+
## Gaps
60+
[What the paper does NOT say that Phase 2 will have to assume. Be explicit.]
61+
62+
## Implications for Phase 2
63+
[What values to use for m, M, p, and which heuristic to model]
64+
```
65+
66+
**Do NOT proceed to Phase 2 if:**
67+
- The paper doesn't report concrete `m:M` or `p` (note gaps explicitly; flag for author follow-up)
68+
- The "overhead" metric is ambiguous (could be any of wall-time / FLOPs / tokens-per-sec)
69+
70+
In that case, stop at Phase 1 and document the gaps.
71+
72+
## Phase 2 — ByteDMD microbenchmark
73+
74+
### Reduced model
75+
76+
Create `experiments/engram-offload/model.py`:
77+
- Toy attention block: `d_model=64`, `seq_len=128`, 1 head
78+
- KV memory bank split into two tiers:
79+
- **resident tier** (size `m`): lives at top of ByteDMD stack (depth ~100-1000)
80+
- **offloaded tier** (size `M`): forced to depth ~`1e6` via padding/unused writes
81+
- Pure Python values so ByteDMD's tracer sees every read. No numpy in the inner loop (numpy escapes the tracer, see `docs/research/bytedmd.md`)
82+
83+
### Three variants, same compute
84+
85+
Create `experiments/engram-offload/run.py`:
86+
87+
1. **`resident`** — all KV entries resident (baseline). Every lookup is shallow.
88+
2. **`offload_naive`** — `M` offloaded entries at depth `1e6`. Every lookup into the offloaded tier pays `sqrt(1e6) = 1000` per byte. No prefetching.
89+
3. **`offload_prefetch`** — paper's heuristic from Phase 1. Predicted-hot entries promoted to the top of the stack each step, reducing how many reads actually pay the deep cost.
90+
91+
All three process the same `tokens × kv_bank` input and produce the same output (deterministic forward pass; only the memory layout differs).
92+
93+
### Measurement
94+
95+
For each variant:
96+
97+
```python
98+
from bytedmd import bytedmd, traced_eval
99+
100+
cost = bytedmd(forward, (tokens, kv_bank))
101+
trace, out = traced_eval(forward, (tokens, kv_bank))
102+
```
103+
104+
Record:
105+
- Total ByteDMD cost
106+
- Cost per token
107+
- Trace depth distribution (histogram of stack depths at each read)
108+
- Wall-time with `time.perf_counter()` for the same forward pass
109+
110+
Save to `experiments/engram-offload/results.json`.
111+
112+
### Findings
113+
114+
Create `docs/findings/engram-offload-bytedmd.md` following `findings/_template.md`. Must include:
115+
116+
- **Hypothesis** (from the paper-read + your prior)
117+
- **Config** (m, M, p, heuristic — reference Phase 1 extraction)
118+
- **Results table** (all three variants: cost, cost-per-token, wall-time, ratios)
119+
- **Classification** (WIN / LOSS / INVALID / INCONCLUSIVE / BASELINE) per the decision thresholds in `docs/tasks/011-engram-offload-bytedmd.md`
120+
- **Analysis** — what worked, what didn't, the surprise
121+
- **Impact on DISCOVERIES.md** (if any) — does this add to the ByteDMD-vs-wall-time story?
122+
123+
## Decision thresholds
124+
125+
From `docs/tasks/011-engram-offload-bytedmd.md`:
126+
127+
| ByteDMD overhead (prefetch vs resident) | Classification | Action |
128+
|---|---|---|
129+
| **2-5×** | WIN for DeepSeek — claim is real | Positive case study; their heuristic genuinely works |
130+
| **>20×** | LOSS — wall-time gaming | Canonical case study for why SutroYaro uses ByteDMD |
131+
| **5-20×** | INCONCLUSIVE | Re-run with larger `M` and varying `p`; bandwidth-hiding advantage should shrink as working set grows |
132+
133+
## Rules
134+
135+
- **DO NOT** modify measurement code: `src/bytedmd/`, `src/harness.py`, `tracker.py`, `cache_tracker.py`, `data.py`, `config.py`, `fast.py`
136+
- **DO NOT** modify `DISCOVERIES.md`, `LAB.md`, `CLAUDE.md`, `CODEX.md`, or any other project configuration. Updates to `DISCOVERIES.md` are added separately via PR after findings review.
137+
- Pure Python in the inner loop. Numpy escapes ByteDMD; see `docs/research/bytedmd.md`.
138+
- All findings go under `docs/findings/` (the mkdocs-published location). Do NOT create files in the root-level `findings/` directory.
139+
- Phase 1 first. Do not start Phase 2 until the paper extraction is written.
140+
141+
## Out of scope
142+
143+
- Re-training or re-implementing the full DeepSeek model. We're testing the metric on a reduced analogue.
144+
- Modeling PCIe/NVMe energy directly. ByteDMD's `sqrt(depth)` is the proxy.
145+
- Modifying the ByteDMD tracer to "handle" offloading more cleverly. The metric is the spec; we're measuring against it, not tuning it.

‎docs/tasks/011-engram-offload-bytedmd.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# Task 11: DeepSeek Engram Offload — ByteDMD Verification
2+
3+
**Priority**: MEDIUM
4+
**Status**: OPEN
5+
**Agent**: unassigned
6+
**Source**: Issue #77 (observation by Andy via Qwen, experiment plan by Yad)
7+
8+
## Context
9+
10+
The DeepSeek Engram paper ("Conditional Memory via Scalable Lookup") claims that 100B parameters can be offloaded to CPU/SSD with <3% inference overhead. From a ByteDMD perspective this claim is suspicious:
11+
12+
- SSD→GPU bandwidth is 20-60× slower than GPU HBM (~16GB/s vs ~1-3TB/s)
13+
- Under ByteDMD, SSD reads live at stack depth ~millions vs ~hundreds for HBM
14+
- Per-byte cost ratio: `ceil(sqrt(1e6)) / ceil(sqrt(1e3)) ≈ 31×`
15+
- If the "3%" is wall-time (async prefetching hiding latency) rather than energy, ByteDMD should expose that
16+
17+
**Relevance to SutroYaro**: this is exactly the "wall-time ≠ energy" confusion that motivates the ByteDMD metric. Either outcome of this task produces valuable output:
18+
19+
- **Claim validated** → heuristic genuinely reduces deep reads; document the pattern as a positive case study
20+
- **Claim is wall-time gaming** → document as the canonical case study for why SutroYaro uses ByteDMD and not throughput
21+
22+
Paper: https://deepseek.ai/blog/deepseek-engram-v4-architecture
23+
24+
## Tasks
25+
26+
### Phase 1 — Paper extraction (prerequisite)
27+
28+
- [ ] Read https://deepseek.ai/blog/deepseek-engram-v4-architecture carefully
29+
- [ ] Extract whether "overhead" is wall-time, FLOPs, or energy
30+
- [ ] Extract concrete `m:M` ratio (resident : offloaded set sizes)
31+
- [ ] Extract lookup frequency `p` (fraction of tokens that hit the offloaded set)
32+
- [ ] Document the prefetch heuristic (predicted-hot entries? random? LRU? model-guided?)
33+
- [ ] Write to `docs/findings/engram-offload-paper-read.md`
34+
35+
Without these specifics, Phase 2 models a generic offload pattern, not Engram specifically.
36+
37+
### Phase 2 — ByteDMD microbenchmark
38+
39+
- [ ] Create `experiments/engram-offload/model.py` — a toy attention block (d_model=64, seq_len=128) with a KV memory bank split into resident vs offloaded tiers. Pure Python so ByteDMD traces all reads.
40+
- [ ] Create `experiments/engram-offload/run.py` with three variants, identical compute:
41+
- `resident`: all weights at small stack depth (baseline)
42+
- `offload_naive`: offloaded bank at depth ~`1e6`, every lookup pays `ceil(sqrt(1e6))=1000` per byte
43+
- `offload_prefetch`: paper's heuristic (promote predicted-hot entries to top each step)
44+
- [ ] Run `bytedmd(forward, (tokens, kv_bank))` for each variant. Record cost, cost-per-token, trace distribution
45+
- [ ] Wall-time control: time the same three on CPU with `time.perf_counter()` to reproduce the "<3%" wall-time claim locally
46+
- [ ] Write findings to `docs/findings/engram-offload-bytedmd.md` following `findings/_template.md`
47+
48+
### Decision thresholds
49+
50+
- **Go** (claim is real): ByteDMD overhead of `offload_prefetch` vs `resident` is **2-5×**. Heuristic genuinely reduces deep reads.
51+
- **No-go** (wall-time gaming): ByteDMD overhead **>20×** while wall-time overhead stays <3%. Write up as case study.
52+
- **Ambiguous**: **5-20×**. Re-run with larger `M` and varying `p`; bandwidth-hiding advantage should shrink as working set grows.
53+
54+
## Out of scope
55+
56+
- Re-training the 100B model. Testing the metric, not the paper.
57+
- PCIe energy models. ByteDMD's `sqrt(depth)` is the proxy.
58+
59+
## References
60+
61+
- Agent prompt: [docs/agent-prompts/engram-offload.md](../agent-prompts/engram-offload.md)
62+
- Paper: https://deepseek.ai/blog/deepseek-engram-v4-architecture
63+
- ByteDMD metric: [docs/research/bytedmd.md](../research/bytedmd.md)
64+
- ByteDMD repo: https://github.com/cybertronai/ByteDMD
65+
- Origin issue: [#77](https://github.com/cybertronai/SutroYaro/issues/77)
66+
- Pattern precedent: Task 9 (Muon review) — [009-muon-review.md](009-muon-review.md)

‎docs/tasks/INDEX.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,3 +14,4 @@ Feedback from Meetings #8-11, Telegram, and Google Docs. Updated 2026-04-19.
1414
| 8 | Pre-experiment plan: DMC comparison, RL env, license | MEDIUM | IN PROGRESS | [008-pre-experiment-plan.md](008-pre-experiment-plan.md) |
1515
| 9 | Muon optimizer literature review (ByteDMD-aware) | MEDIUM | DONE | [009-muon-review.md](009-muon-review.md) |
1616
| 10 | ASI-Evolve paper review — autonomous loop lessons | HIGH | IN PROGRESS | [010-asi-evolve-integration.md](010-asi-evolve-integration.md) |
17+
| 11 | DeepSeek Engram offload — ByteDMD verification | MEDIUM | OPEN | [011-engram-offload-bytedmd.md](011-engram-offload-bytedmd.md) |

0 commit comments

Comments
 (0)