Every template I build ends up unusable after a few minutes — memfile.header is missing on disk. Am I doing something wrong?

[zegging]

Every template I build ends up unusable after a few minutes — memfile.header is missing on disk. Am I doing something wrong?

Hi! I'm setting up a self-hosted E2B locally for the first time and I'm stuck. Every template I build starts working
immediately but then permanently breaks a few minutes later, and I don't understand what I'm doing wrong. I dug into
the orchestrator logs and the on-disk template storage and found something that looks suspicious, but I don't know the
code well enough to say whether it's a real bug or a misconfiguration on my end. Any pointers would be very appreciated.

Environment

• e2b-infra: commit 6c512329de91c3e8a3b49be8d9f72e61d794fcee on main (git describe: 2026.24-37-g6c512329d)
• Host OS: Ubuntu 26.04 LTS (Resolute Raccoon), kernel 7.0.0-22-generic, x86_64
• Go: go1.26.0 linux/amd64
• Node: v22.22.1
• Postgres: 16 (from apt), reachable at localhost:5432 with the built-in postgres user
• Redis: localhost:6379
• ClickHouse: localhost:9000
• Storage provider: Local (all built artefacts on the local filesystem — no GCP/AWS bucket)
• Firecracker used at runtime: v1.14.1_431f1fc (from packages/fc-versions/builds/)
• Guest kernel: vmlinux-6.1.158 (from packages/fc-kernels/)
• envd: 0.6.6 (from packages/envd/bin/envd)
• Client SDK: @qunhe/e2b@2.31.1. This is a fork of @e2b/sdk@2.31.1 (source: https://github.com/zegging/E2B)
  that only adds a sandboxOrigin option for rewriting sandbox envd traffic to a custom origin. The template build
  API (Template.build(...)) is unchanged, so the exact same reproduction should work with the upstream SDK.

How I'm running the stack

Everything on one box. Three long-running services, all launched from
/home/tunan/src/e2b-infra. The environment for each comes from that package's .env.local.

orchestrator (as root, needs /dev/kvm + firecracker + NBD):

cd /home/tunan/src/e2b-infra/packages/orchestrator
sudo -E env $(grep -v '^#' .env.local | xargs) \
    NODE_ID=$(hostname) ./bin/orchestrator

Effective env (from packages/orchestrator/.env.local):

ARTIFACTS_REGISTRY_PROVIDER=Local
STORAGE_PROVIDER=Local
LOCAL_BUILD_CACHE_STORAGE_BASE_PATH=./tmp/local-build-cache
LOCAL_TEMPLATE_STORAGE_BASE_PATH=./tmp/local-template-storage
SANDBOX_CACHE_DIR=./tmp/sandbox-cache-dir
SNAPSHOT_CACHE_DIR=./tmp/snapshot-cache
ORCHESTRATOR_BASE_PATH=./tmp/
ORCHESTRATOR_SERVICES=orchestrator,template-manager
FIRECRACKER_VERSIONS_DIR=../fc-versions/builds
HOST_KERNELS_DIR=../fc-kernels
HOST_ENVD_PATH=../envd/bin/envd
NBD_POOL_SIZE=16
CLICKHOUSE_CONNECTION_STRING=clickhouse://clickhouse:clickhouse@localhost:9000/default
REDIS_URL=localhost:6379
LOGS_COLLECTOR_ADDRESS=http://localhost:30006
OTEL_COLLECTOR_GRPC_ENDPOINT=localhost:4317
ENVIRONMENT=local
DEFAULT_PERSISTENT_VOLUME_TYPE=test-volume-type
PERSISTENT_VOLUME_MOUNTS=test-volume-type:./.data/test-volume

client-proxy (regular user):

cd /home/tunan/src/e2b-infra/packages/client-proxy
env $(grep -v '^#' .env.local | xargs) \
    NODE_ID=$(hostname) ./bin/client-proxy

api (regular user):

cd /home/tunan/src/e2b-infra/packages/api
env $(grep -v '^#' .env.local | xargs) ./bin/api

ss -tln confirms:

LISTEN 0 4096 *:3000    (api)
LISTEN 0 4096 *:3002    (client-proxy)
LISTEN 0 4096 *:3003    (client-proxy)
LISTEN 0 4096 *:5008    (orchestrator gRPC + /upload HTTP via cmux)

The api-server startup log confirms exactly one node and template-manager, healthy:

API internal status  nodes_count=1 nodes=[{id:local,sandboxes:0,status:ready}]
                     template_managers_count=1
                     template_managers=[{cluster_id:...,node_id:local,status:Healthy}]

How I'm creating templates

Straight from the JS SDK. Here are the two scripts I've tried. Both fail the same way once the sandbox is created a
few minutes after Template.build returns.

Script A — build-template.mjs (2 vCPU / 2048 MB, adds JDK 8 + Node 20 on top of base):

import 'dotenv/config'
import { Template } from '@qunhe/e2b'

const template = Template()
  .fromTemplate('base')
  .aptInstall(['curl', 'git', 'ca-certificates', 'gnupg'])
  .runCmd([
    'mkdir -p /etc/apt/keyrings',
    'curl -fsSL https://packages.adoptium.net/artifactory/api/gpg/key/public | gpg --dearmor -o /etc/apt/keyrings/adoptium.gpg',
    'echo "deb [signed-by=/etc/apt/keyrings/adoptium.gpg] https://packages.adoptium.net/artifactory/deb bookworm main" > /etc/apt/sources.list.d/adoptium.list',
    'apt-get update',
    'apt-get install -y temurin-8-jdk',
  ], { user: 'root' })
  .runCmd([
    'curl -fsSL https://deb.nodesource.com/setup_20.x -o /tmp/nodesource.sh',
    'bash /tmp/nodesource.sh',
    'apt-get install -y nodejs',
    'rm -f /tmp/nodesource.sh',
  ], { user: 'root' })

const info = await Template.build(template, 'llzz-dev', {
  cpuCount: 2,
  memoryMB: 2048,
  onBuildLogs: (e) => console.log(`[${e.level}] ${e.message}`),
})
console.log('templateId =', info.templateId, 'buildId =', info.buildId)

Script B — build-small.mjs (1 vCPU / 1024 MB, just apt-installs a couple of things):

import 'dotenv/config'
import { Template } from '@qunhe/e2b'

const template = Template()
  .fromTemplate('base')
  .aptInstall(['curl', 'git'])

const info = await Template.build(template, 'llzz-small', {
  cpuCount: 1,
  memoryMB: 1024,
  onBuildLogs: (e) => console.log(`[${e.level}] ${e.message}`),
})
console.log('templateId =', info.templateId)

Both scripts run to completion without any error. Template.build returns cleanly and prints a real buildId.

.env used by the SDK:

E2B_API_KEY=<a real key>
E2B_ACCESS_TOKEN=<a real token>
E2B_API_URL=http://10.10.21.131:3000
E2B_DOMAIN=e2b.app
E2B_SANDBOX_ORIGIN=http://10.10.21.131:3002

Symptom (what actually happens)

Signal 1 — build looks successful. In Postgres, right after Template.build returns:

SELECT alias, env_id FROM env_aliases WHERE alias IN ('base','llzz-dev','llzz-small');
   alias    |        env_id
------------+----------------------
 base       | gg1xl4lhgm0cht8rjcn9
 llzz-dev   | vlglbzscrd6fbi49syef
 llzz-small | 7ne9lt6whcnaox6kwdo1

SELECT id, env_id, status FROM env_builds WHERE status='uploaded';
                  id                  |        env_id        |  status
--------------------------------------+----------------------+----------
 c7c8b8c3-3fe5-4f40-8d4a-6bad22dc9d2b | gg1xl4lhgm0cht8rjcn9 | uploaded   <-- base
 5958e427-aff3-45fd-876f-f54904802864 | vlglbzscrd6fbi49syef | uploaded   <-- llzz-dev
 13b04d21-7ab4-4312-846e-14fadb10cc7c | 7ne9lt6whcnaox6kwdo1 | uploaded   <-- llzz-small

All three builds show status='uploaded'. Looks good.

Signal 2 — sandbox creation works for a few minutes after build.
Within roughly 3 minutes of Template.build returning, I can create sandboxes from my new template and they behave
normally. In packages/orchestrator/tmp/e2b-local-runtime/logs/orchestrator.log I see the sandboxes come up cleanly
with adding sandbox to map ... build.id=5958e427..., and their Firecracker snapshot loads succeed against local paths
under ./tmp/template/<buildId>/cache/<cacheId>/snapfile.

Signal 3 — after that grace window, every Sandbox.create for my templates fails permanently.
Test I ran to confirm the pattern:

import { Sandbox } from '@qunhe/e2b'
for (const tpl of ['base', 'llzz-small', 'llzz-dev']) {
  try {
    const t0 = Date.now()
    const sbx = await Sandbox.create(tpl, { timeoutMs: 60_000 })
    console.log('OK  ', tpl, 'id=', sbx.sandboxId, '(' + (Date.now()-t0) + 'ms)')
    await sbx.kill()
  } catch (e) { console.log('FAIL', tpl, e.message) }
}

Output:

OK   base         id= i9o7rhcmct0y5ibvyaqb2 (248ms)
FAIL llzz-small   500: Failed to place sandbox
FAIL llzz-dev     500: Failed to place sandbox

Only base works. All templates I built myself fail — including a minimal one with no dockerfile magic.

Server-side error chain

In packages/api/.../logs/api.log:

2026-07-08T15:28:26 ERROR Failed to create sandbox
  sandbox.id: ievncbmfbpmdnxsd12us8
  node.id: local
  error: "[FailedPrecondition] sandbox files for 'ievncbmfbpmdnxsd12us8' not found"
2026-07-08T15:28:26 WARN  error when creating instance
  error: "failed to place sandbox: no nodes available"
2026-07-08T15:28:26 ERROR /sandboxes  status=500  error="Failed to place sandbox"

Same trace in orchestrator.log, one layer down:

2026-07-08T15:28:26 WARN  sandbox files not found
  sandbox.id: ievncbmfbpmdnxsd12us8
  error: "failed to get snapfile: failed to fetch snapfile: NEW STORAGE failed to write to file: object does not exist"
2026-07-08T15:28:26 WARN  SandboxService/Create/unary [FailedPrecondition]: finished call
  grpc.code: FailedPrecondition
  grpc.error: "rpc error: code = FailedPrecondition desc = sandbox files for 'ievncbmfbpmdnxsd12us8' not found"

Just before the sandbox files not found, the log floods with many warnings that all look like this:

WARN offset is beyond the end of mapping, but normalize fix is not applied
     offset:       2107637760    (~2.0 GB, matches ram_mb=2048 for llzz-dev)
     mappingOffset: 0
     mappingEnd:   241172480     (~230 MB, matches physical memfile size on disk)
     build.id:     5958e427-aff3-45fd-876f-f54904802864
WARN mapped length is negative, but normalize fix is not applied
     offset:       2107637760
     mappedLength: -1866465280
     build.id:     5958e427-aff3-45fd-876f-f54904802864

Firecracker is asking for memory pages at offsets around 2 GB, but the orchestrator's mapping thinks the memfile ends
at ~230 MB.

What I found on disk — this is what I don't understand

LOCAL_TEMPLATE_STORAGE_BASE_PATH resolves to
/home/tunan/src/e2b-infra/packages/orchestrator/tmp/local-template-storage (relative to the orchestrator's cwd).
Comparing what's inside each build's directory:

sudo ls -la .../local-template-storage/c7c8b8c3-3fe5-4f40-8d4a-6bad22dc9d2b/ — this is base, works:

memfile             159 383 552   Jun 23 16:42
memfile.header              264   Jun 23 16:42     ← present
metadata.json             1 243   Jun 23 16:42
rootfs.ext4           6 520 832   Jun 23 16:42
rootfs.ext4.header       44 584   Jun 23 16:42
snapfile                 29 464   Jun 23 16:42

sudo ls -la .../local-template-storage/5958e427-aff3-45fd-876f-f54904802864/ — this is llzz-dev, broken:

memfile             241 172 480   Jul  7 17:48
                                                 ← memfile.header MISSING
metadata.json             1 342   Jul  7 17:48
rootfs.ext4          11 546 624   Jul  7 17:48
rootfs.ext4.header      101 904   Jul  7 17:48
snapfile                 29 464   Jul  7 17:48

sudo ls -la .../local-template-storage/13b04d21-7ab4-4312-846e-14fadb10cc7c/ — this is llzz-small, broken:

memfile             159 383 552   Jul  8 11:14
                                                 ← memfile.header MISSING
metadata.json             1 278   Jul  8 11:14
rootfs.ext4           6 373 376   Jul  8 11:14
rootfs.ext4.header       48 384   Jul  8 11:14
snapfile                 16 487   Jul  8 11:14

Both templates I built are missing only memfile.header; everything else is there. The mappingEnd: 241172480 in
the warnings above is exactly the physical size of my llzz-dev memfile on disk, which lines up with the theory that
orchestrator has no diff header to expand it into a virtual 2 GB mapping.

What I've tried to make sense of this

1. Grepped orchestrator.log for anything upload-related — I never see an ERROR level "template layer snapshot upload
   failed" or "error uploading snapshot", and I never see a snapshot finished uploading successfully line for the
   build IDs of my custom templates (only for base sandbox pauses).

2. grep '/upload' in the orchestrator log for the whole session (~15 days uptime) returns 0 HTTP /upload
   requests — the local upload endpoint at http://localhost:5008/upload is registered (I can curl it and get 403
   as expected because I don't have a token), but nothing in the log ever hits it, even for builds that "succeeded".

3. Manually building even the trivial script B (aptInstall(['curl','git']) on top of base) reproduces the same
   missing-memfile.header outcome, so it doesn't seem to depend on what my dockerfile does.

4. During the llzz-dev build I also see some NBD read errors reading the base template's memfile / rootfs:

   ERROR nbd backend read failed
     error: "error reading from device: failed to slice cache at 869437440-869441536:
             fetch failed: failed to open range reader at 868220928: object does not exist"
   

   which makes me wonder whether base itself is also partially incomplete on disk, and I've just been lucky that
   sandboxes based on base never touch the missing ranges. But I don't know the layout well enough to say.

What I think might be happening, but I'm not sure

Reading packages/orchestrator/pkg/sandbox/build_upload_v3.go and build_upload_v4.go, the memfile body and the
memfile diff header are uploaded from two independent goroutines inside an errgroup:

// build_upload_v3.go
eg.Go(func() error {
    h, err := u.snap.MemorySnapshot.DiffHeader.WaitWithContext(egCtx)
    if err != nil { return fmt.Errorf("wait memfile diff header: %w", err) }
    if h == nil { return nil }
    return storeHeaderWithMetrics(egCtx, u.store, u.paths.MemfileHeader(),
        uploadFileMemfileHeader, finalizeV3(h), storage.WithMetadata(u.objectMetadata))
})

eg.Go(func() error {
    if memfilePath == "" { return nil }
    info, err := os.Stat(memfilePath)
    if err != nil { return fmt.Errorf("memfile stat: %w", err) }
    _, _, err = storage.UploadFramed(egCtx, u.store, u.paths.Memfile(),
        storage.MemfileObjectType, memfilePath, meta)
    ...
})

My best guess (as an outsider!) is that in my setup MemorySnapshot.DiffHeader.WaitWithContext is resolving to h == nil, so the header-upload goroutine returns nil without doing anything, while the memfile-body goroutine finishes
normally — leaving on-disk state exactly matching what I observe (memfile present, memfile.header absent). The build
then reports success and TemplateManager.SetFinished in packages/api/internal/template-manager/template_status.go
writes status='uploaded' regardless. The template cache keeps the freshly-built snapshot in memory for a while, which
would also explain why Sandbox.create works for a few minutes after the build.

But I don't understand:

• Under what conditions is DiffHeader legitimately nil for a non-filesystem-only snapshot with a real memfile
  diff? Is this a state that's supposed to be reachable in a healthy build?
• If this is a legitimate state, is my configuration wrong somehow such that I'm producing "header-less" snapshots?
• If it isn't a legitimate state, is there a known upstream setup step (some manual bootstrap for Local storage
  provider?) that I've skipped?
• Related: is it expected that base (which was bundled/preloaded on this box before I started, not built by me)
  produces the NBD "object does not exist" errors above when it's used as a parent layer?

Anything else that might be relevant

• .pi/ and everything else in /home/tunan/src/e2b-infra/packages/orchestrator/tmp is 700 root:root; I only get in
  with sudo.
• I've never restarted the orchestrator since the base template was populated (its PID has been running for ~15
  days). The build IDs of both broken templates were produced by this orchestrator instance, though.
• The 5008 upload endpoint responds correctly to curl -X PUT with a 403 (bad token) as expected.
• I get the same behaviour whether I use Template.build(template, 'llzz-dev', {...}) from my Node script or invoke
  the same build via the CLI — so I don't think it's SDK-specific. (The SDK is a fork but only patches sandbox envd
  routing.)

Happy to grab any additional trace / metadata I've missed if it would help. Thanks for reading!

[AdaAibaby]

@zegging Please merge PR #3251 first. When you reproduce the issue, share the full error logs — they will help us trace the root cause precisely.

[zegging]

Thanks for taking a look!

Update: re-ran the whole thing end-to-end with a fresh minimal build and captured raw logs at exact byte offsets. Everything below is on the same box, same commit 6c512329d, upstream e2b@2.9.0 SDK (no fork this time).

Repro

import { Template, Sandbox } from 'e2b'

const alias = 'repro-' + Date.now()
const t = Template().fromTemplate('base').aptInstall(['curl'])

const info = await Template.build(t, { alias, cpuCount: 1, memoryMB: 1024 })
// (1) immediate
await Sandbox.create(alias, { timeoutMs: 20000 })
// (2) 5 min later
await new Promise(r => setTimeout(r, 5 * 60 * 1000))
await Sandbox.create(alias, { timeoutMs: 20000 })

Findings

Build reports success. Template.build returned in 37s with buildId=d89d84d7-0e89-457d-bd8d-a6861ebb34e5. Postgres agrees:

env_builds:  d89d84d7-... | 6z3s4hpxxr3pk5x6s400 | uploaded | 2026-07-10 13:51:54.404793+00

memfile.header is missing on disk within seconds of Template.build returning — so it is the build that produces the header-less template, not something that happens later:

$ sudo ls -la .../local-template-storage/d89d84d7-0e89-457d-bd8d-a6861ebb34e5/
-rw-r--r-- 1 root root 161480704 memfile
                                             # memfile.header missing
-rw-r--r-- 1 root root      1299 metadata.json
-rw-r--r-- 1 root root   6881280 rootfs.ext4
-rw-r--r-- 1 root root     47744 rootfs.ext4.header
-rw-r--r-- 1 root root     16487 snapfile

rootfs.ext4.header is present on every build, only memfile.header is absent — the discriminator that decides "produce a diff header" is per-file.

Clarification on the grace window. Both Sandbox.create calls fail immediately:

attempt	when	result	latency
1	right after build returns	500: Failed to place sandbox	161 ms
2	5 min later	500: Failed to place sandbox	561 ms

The "few minutes of working sandboxes" I described earlier was almost certainly a warm in-memory cache from a different template — not the freshly built one. The freshly built one is broken from t=0.

Error chain (top-down, full stack traces in the attached logs):

API      POST /sandboxes    → HTTP 500 "Failed to place sandbox"
         └─ failed to place sandbox: no nodes available
              └─ [FailedPrecondition] sandbox files for '<sbxId>' not found

orchestrator  SandboxService/Create/unary  build.id=d89d84d7-...
              └─ WARN sandbox files not found
                     error: failed to get rootfs: failed to create rootfs storage:
                            failed to get object size: object does not exist
                     .../pkg/server/sandboxes.go:254

Next steps

i will merge PR #3251 and try again.

Attached (raw, unedited)

• state.json, on-disk-after-build.txt, db-after-build.txt
• api-build.log / orchestrator-build.log — build phase
• api-sbx1-immediate.log / orchestrator-sbx1-immediate.log — create 1
• api-sbx2-after-5min.log / orchestrator-sbx2-after-5min.log — create 2
• api-full.log / orchestrator-full.log — three phases stitched
• repro2.mjs, script-stdout.log

end-to-end-logs.zip