[FVM] Long-lived derived data (programs) cache can rot — add bounded entry lifetime and staleness detection

[janezpodhostnik]

Problem

Parsed and checked Cadence contracts are kept in the derived data cache (fvm/storage/derived/), alongside derived execution parameters. The cache is intentionally long-lived since contracts change infrequently — but entry lifetime is effectively unbounded: on each new block, DerivedDataTable.NewChildTable copies all parent entries into the child table, reusing the same in-memory *runtime.Program objects. An entry survives until explicit invalidation or node restart — potentially the node's entire uptime.

Invalidation (fvm/environment/derived_data_invalidator.go) is the sole freshness mechanism. If it has a gap, or a long-lived in-memory object silently degrades, the cached data rots and FVM execution produces incorrect results — with no bound on how long the bad entry keeps being served.

The impact of a rotted entry differs by execution path:

• Block execution: VNs re-derive programs from scratch per chunk (module/chunks/chunkVerifier.go), so a rotted EN cache should fail verification — a liveness incident rather than wrong sealed state, but still an incident.
• Script execution: the EN query executor and the AN/ON program cache (--program-cache-size, experimental) serve results directly to users with no verification safety net — stale results here are silent.

Findings from initial investigation

• Eviction is consensus-neutral. Cache hits replay the stored ExecutionSnapshot onto the transaction state, so hit vs miss yields identical execution results (VNs already run every chunk cold and match warm-cache EN results). Dropping entries can only affect performance.
• Naive per-entry random eviction is unsafe. Per the invariant documented in fvm/storage/derived/table.go, the table must always return the same *Program object for a key, because dependents embed Cadence types from that specific object. Eviction must drop dependency-closed sets — the semantics ProgramInvalidator already implements for contract updates.
• Hashing for detection is feasible for inputs, partial for objects. Storing a hash of the (registerID, value) pairs read during derivation would let a sampled background checker re-read those registers against committed state — detecting invalidation gaps independently of the invalidator code. Hashing the cached object itself only covers the AST (canonical JSON); the Elaboration is a pointer graph with no canonical serialization.

Proposed direction

1. Bounded lifetime via periodic full flush (recommended starting point): with small probability per block — or every N blocks / T hours — create the child DerivedBlockData empty instead of copying the parent. Trivially safe: identical to the routinely-exercised post-restart state; cost is a brief warm-up.
2. Input-hash staleness detection: store a hash of read register values per entry; background-sample and verify against committed state, invalidating (dependency-closed) and alerting on mismatch.
3. Operational hygiene: an admin command to flush DerivedChainData on a live node, and a cache entry-age metric (today only hit/miss counters exist).

Relevant code

• fvm/storage/derived/derived_chain_data.go — block-level LRU
• fvm/storage/derived/table.go — NewChildTable carry-forward, OCC validation, object-identity invariant
• fvm/environment/derived_data_invalidator.go — invalidation rules (dependency-closed)
• fvm/storage/snapshot/execution_snapshot.go — per-entry derivation snapshot (ReadSet has no values today)
• module/chunks/chunkVerifier.go — VN cold-cache re-execution (the safety net for block execution)