fix(orchestrator): memfile.header not written in Local storage mode — templates break after cache eviction

[AdaAibaby]

Problem

When using STORAGE_PROVIDER=Local, newly built templates work for a few minutes then permanently fail with:

object does not exist
failed to get snapfile: NEW STORAGE failed to write to file: object does not exist
[FailedPrecondition] sandbox files for '<id>' not found

Root cause

In build_upload_v3.go / build_upload_v4.go, the memfile header and body are uploaded in separate goroutines:

eg.Go(func() error {
    h, err := u.snap.MemorySnapshot.DiffHeader.WaitWithContext(egCtx)
    if err != nil { return ... }
    if h == nil { return nil }   // ← returns without writing header
    return storeHeaderWithMetrics(...)
})

In the Local storage path, DiffHeader.WaitWithContext resolves to nil, so the header goroutine exits cleanly without writing memfile.header. The body goroutine writes memfile successfully. The build is then marked uploaded even though the snapshot is incomplete.

The orchestrator's in-memory snapshot cache serves sandboxes correctly for a few minutes. Once the cache entry is evicted, every subsequent Sandbox.create hits local storage, finds no memfile.header, and fails with FailedPrecondition.

Expected behaviour

memfile.header must be written to local storage on every successful template build that produces a memory snapshot.

Observed on-disk state

• base (pre-bundled): memfile ✓, memfile.header ✓, rootfs ✓, snapfile ✓
• user-built template: memfile ✓, memfile.header missing, rootfs ✓, snapfile ✓

Reported by

#3226

[jakubno]

Hey @AdaAibaby , that sounds like a regression. Are you willing to contribute a fix?

[AdaAibaby]

Hey @AdaAibaby , that sounds like a regression. Are you willing to contribute a fix?

Will fix this, submitted PR #3251 to resolve the issue.