fix(langgraph): hydrate subgraph delta channels with the caller-resolved saver

[Elior Nataf Lackritz (eliornl)]

Fixes #8470

Reported by gururafiki, with a self-contained InMemorySaver repro and the observation that non-delta channels in the same namespace hydrate fine, which is what makes the failure silent.

_prepare_state_snapshot re-derived the saver from self.checkpointer, which is None for a subgraph, so every DeltaChannel in a nested subgraph hydrated empty on read, silently.

Why the old resolution was wrong

A DeltaChannel stores no value in channel_values, so reconstructing it means walking ancestors and replaying their writes, which needs a saver. A subgraph has none of its own: it borrows the parent's through CONFIG_KEY_CHECKPOINTER at read time, exactly as the four public readers already resolve it a few lines above the call.

_prepare_state_snapshot ignored that and went back to self.checkpointer. With no saver the walk never ran, and the channel fell through to from_checkpoint(MISSING): an empty value, indistinguishable from a channel that was never written, raising nothing. Non-delta channels in the same namespace hydrate fine, so the payload looks healthy.

Recovering it inside the function isn't possible: get_state_history passes checkpoint_tuple.config, the checkpointer's stored config, which never carries a live saver. The caller is the only place the resolved saver exists, so the caller has to pass it.

How I verified it

tests/test_delta_channel_subgraph.py adds 12 tests. 8 fail on main, 4 pass. The 4 that pass are controls for modes that stay unchanged (the two root-graph cases, a stateless subgraph, a completed subgraph). The 8 failures, by surface:

surface	on main
get_state / aget_state on a subgraph ns	{'msgs': []}
get_state_history / aget_state_history	every snapshot empty
two levels of nesting	{'msgs': []}
get_state(subgraphs=True) task state, subgraph interrupted	{'msgs': []}
update_state / aupdate_state	['manual'], prior history destroyed

Tests use InMemorySaver: the defect is in Pregel's saver resolution rather than in any checkpointer implementation, so the storage backend is irrelevant to the reproduction.

Full suite 1968 -> 1980 passed, 4 skipped, exactly the 12 new tests. make format, lint_package, lint_tests clean; check_sdk_methods.py passes.

Two things worth a closer look in review

1. This fixes a write path as well as a read path, and both are load-bearing. bulk_update_state / abulk_update_state carried the same expression, and that half is worse than a misleading read: when the update reaches the channel's snapshot cadence, create_checkpoint persists a _DeltaSnapshot built from the empty-hydrated channel, losing history on disk. I checked whether it could be split: reverting only the two bulk_update_state hunks leaves test_subgraph_update_state_preserves_delta_channel_history and its async twin failing while the other 10 pass. Splitting would ship a fix that still corrupts history on write.

The write-path tests use snapshot_frequency=2, which is what makes the update reach the cadence and force a snapshot blob. At the default 1000 nothing is written, the loss stays latent, and the assertion would hold either way.

2. prepare_next_tasks in the same two methods still receives self.checkpointer, and I left it. It is inert on the read path: _algo.py:915 builds the task config as checkpointer or configurable.get(CONFIG_KEY_CHECKPOINTER), so a subgraph's None already falls back to the config value, and PregelTask carries no config field, so that config never reaches a reader. Changing it would be cosmetic, but I'll make it symmetric if you'd rather not leave two different expressions side by side.

Note on the signature

saver is a required keyword argument rather than a defaulted one. A silent fallback is what caused this bug, and both methods are private with no other callers, so there is nothing to stay compatible with. recurse stays defaulted because None is meaningful there (do not resolve subgraph task states); for saver, None is only ever a bug.

Net -6 lines of logic: passing an already-narrowed saver removes the isinstance dance at every site.

[open-swe]

✅ Open SWE Review: No issues found

Open SWE reviewed this PR and found no potential bugs to report.

Open in Web • View Open SWE trace

[Sydney Runkle (sydney-runkle)]

a few questions -- can we find any related bugs in deepagents?
is a checkpointer always none for subgraph? i think there are many options for how we can add: https://docs.langchain.com/oss/python/langgraph/use-subgraphs#subgraph-persistence...

[Sydney Runkle (sydney-runkle)]

basically i'm not actually sure this is incorrect, i think it might just be a feature of how subgraphs manage state, worth reading the above

[Elior Nataf Lackritz (eliornl)]

Good questions, and the doc changed my framing, so let me redo this properly.

On whether it's a feature of subgraph state. I went through the three modes the doc describes, using the read path it documents (get_state(config, subgraphs=True).tasks[0].state) rather than the raw checkpoint_ns read the issue uses. Setup is one subgraph state holding two channels written by the same nodes, one a DeltaChannel and one a plain operator.add list.

Stateless (checkpointer=False) gives nothing at all, and after a run completes there's no task state for either channel. You're right about both, they're subgraph behaviour rather than delta replay, and this PR leaves them alone. I've added tests pinning them so they stay that way.

The two that concern me are per-invocation and per-thread, during an interrupt. Both give:

plain (operator.add) : ['a1']
delta (DeltaChannel) : []

Same call, same instant, same state object. A subgraph-state rule would empty both. It's only the delta channel because that's the one type which stores no value in channel_values at all, so it has to walk ancestors and replay writes, and that needs a saver. _prepare_state_snapshot was passing self.checkpointer.

On whether a checkpointer is always None. It isn't, and per-thread is the case I'd most like your read on. checkpointer=True is an explicit opt-in to use the parent's checkpointer, and it also resolved to no saver, because isinstance(True, BaseCheckpointSaver) is False.

On deepagents. I cloned it at 0.7.4, built a parent with one subagent using their own fake chat model from the unit tests so there's no network, ran a turn, and dumped every namespace. Subagents do checkpoint under tools:<task_id>, and the bug is real in those bytes: files reads back {} while messages in the same read comes back with all four messages, and a control read by a graph that owns the saver returns the file contents fine.

But nothing that ships reaches that read. A subagent lives in the task tool's closure rather than as a node, so get_subgraphs(recurse=True) is empty and get_state(checkpoint_ns="tools:...") raises Subgraph tools not found, identically before and after this change. I had to hold the subagent Pregel directly to trigger anything. So on deepagents this is latent, and this PR moves nothing user-visible there.

Related issues point the same way. #4818 is a user doing exactly that read, closed by mdrxy with "the tools:<uuid> checkpoints are internal". #2629 closed by ccurme on the same grounds. And your #777 treats fetchable subagent history as something we haven't built.

Where I land. The documented limitation covers subgraphs reached through indirection, and its symptom is a hard error. This PR's case is a subgraph added as a node, which the docs list as supported, where you get a well-formed snapshot with one channel silently empty next to a correct one. Those seem like different things to me. I also couldn't find anywhere we'd decided the empty case is fine: no test in libs/langgraph, the conformance spec, or langgraph-api ever puts a DeltaChannel outside checkpoint_ns: "", so it's untested rather than asserted. And #7127/#7128 are open reports of subgraph state being unreadable in per-thread mode.

So I'd still merge, but deepagents isn't the argument for it. If that were the only case I'd close this. What keeps me on it is the node-added subgraph plus the write path, where update_state persists the empty hydration and loses history on disk.

[gururafiki]

Hi! Thanks for opening PR and for review. I have experienced related problem in deepagents. Here is the opened issue for it: langchain-ai/deepagents#5136

[Elior Nataf Lackritz (eliornl)]

Thanks, and good to see #5136 written up. Worth separating the two though, since they look alike from the outside.

#5136 is the tool-indirection case: the subagent lives in the task tool's closure, so LangGraph can't statically discover it and the read raises Subgraph tools not found. That's documented as unsupported, and this PR doesn't change it. I checked separately while looking into deepagents, and the error is identical before and after this change.

This PR is the case where the subgraph is statically discoverable, added as a node, which the docs list as supported. There you get no error at all, just a well-formed snapshot with a DeltaChannel silently empty next to a correct plain channel. Same asymmetry you saw in the stored deepagents data, but reachable through a supported path.

[Dinesh Yadav (dineshyadav03)]

Read through the diff and the linked repro. Confirmed the root cause: get_state/aget_state already resolve checkpointer correctly via CONFIG_KEY_CHECKPOINTER (falling back to self.checkpointer), but _prepare_state_snapshot/_aprepare_state_snapshot discarded that local and re-read self.checkpointer directly, which is None for any subgraph. Since DeltaChannel has no value in channel_values and needs a saver to replay ancestor writes, that produced an empty-but-valid-looking snapshot instead of an error. Good catch pulling in bulk_update_state/abulk_update_state too — that one persists the empty-hydrated channel as a _DeltaSnapshot blob once snapshot_frequency cadence hits, i.e. actual on-disk history loss.

Question: prepare_next_tasks in the same two methods still gets self.checkpointer and is called inert because _algo.py falls back to CONFIG_KEY_CHECKPOINTER internally. Is that fallback covered by a test, or only reasoned about? Given this whole bug was exactly this kind of "should be equivalent" assumption, I'd rather see the two expressions unified even if cosmetic today.

Making saver a required kwarg rather than defaulting to self.checkpointer is the right call. Tests are thorough. Approve.