Skip to content

Reading a nested subgraph's state hydrates every DeltaChannel empty #8470

Description

@gururafiki

Summary

Reading a nested subgraph's state returns every DeltaChannel empty, silently. Non-delta channels in the same namespace hydrate correctly, so nothing errors and nothing looks wrong — the caller just sees an empty channel and cannot tell it from a subgraph that genuinely never wrote one.

This bites anyone inspecting run history per subgraph: messages on an agent that opts into delta storage is exactly the channel you want, and it always comes back [].

Repro (self-contained, InMemorySaver)

from typing import Annotated
from typing_extensions import TypedDict
from langgraph.channels.delta import DeltaChannel
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.graph import START, END, StateGraph

def _append(state, writes):
    out = list(state or [])
    for w in writes:
        out.extend(w if isinstance(w, list) else [w])
    return out

class S(TypedDict, total=False):
    msgs: Annotated[list, DeltaChannel(_append)]

child = StateGraph(S)
child.add_node("a", lambda s: {"msgs": ["a1"]})
child.add_node("b", lambda s: {"msgs": ["b1", "b2"]})
child.add_edge(START, "a"); child.add_edge("a", "b"); child.add_edge("b", END)

parent = StateGraph(S)
parent.add_node("child", child.compile())
parent.add_edge(START, "child"); parent.add_edge("child", END)

app = parent.compile(checkpointer=InMemorySaver())
cfg = {"configurable": {"thread_id": "t1"}}
app.invoke({}, cfg)

child_ns = next(
    t.state["configurable"]["checkpoint_ns"]
    for snap in app.get_state_history(cfg)
    for t in snap.tasks
    if t.name == "child" and isinstance(t.state, dict)
)

print(app.get_state(cfg).values["msgs"])                      # ['a1', 'b1', 'b2']  ✅
print(app.get_state({"configurable": {"thread_id": "t1",
                     "checkpoint_ns": child_ns}}).values)      # {'msgs': []}        ❌

Expected: the child namespace reports ['a1', 'b1', 'b2'].
Actual: [].

Affects get_state, aget_state, get_state_history and aget_state_history.

Cause

A subgraph obtained through get_subgraphs() was compiled without a checkpointer — the parent supplies one via CONFIG_KEY_CHECKPOINTER at read time, which is how the public readers already resolve it (pregel/main.py):

checkpointer = ensure_config(config)[CONF].get(CONFIG_KEY_CHECKPOINTER, self.checkpointer)

But _prepare_state_snapshot then hydrates channels using self.checkpointer alone, so saver is None for any nested subgraph:

channels, managed = channels_from_checkpoint(
    self.channels, saved.checkpoint,
    saver=self.checkpointer if isinstance(self.checkpointer, BaseCheckpointSaver) else None,
    config=saved.config,
)

channels_from_checkpoint needs that saver to call get_delta_channel_history and replay a DeltaChannel's ancestor writes. Without it, the channel falls through to from_checkpoint(MISSING) — empty, no error (pregel/_checkpoint.py):

if delta_channels and saver is not None and config is not None:
    histories = saver.get_delta_channel_history(config=config, channels=delta_channels)
...
ch = spec.from_checkpoint(checkpoint["channel_values"].get(k, MISSING))

Recovering it from the config isn't possible either: get_state_history passes checkpoint_tuple.config — the checkpointer's stored config, which never carries CONFIG_KEY_CHECKPOINTER.

Suggested fix

Pass the already-resolved saver into _prepare_state_snapshot / _aprepare_state_snapshot from the four public readers, preferring it over self.checkpointer. +26/−4, read path only.

I have this working with regression tests (subgraph history, subgraph get_state, async history, plus a root-graph control) — the subgraph tests fail on main and pass with the change, and the full suite shows a byte-for-byte identical failure set before and after. Happy to open the PR once this is approved; it was auto-closed for not linking an issue.

How it was found

Building a run-inspection UI on POST /threads/{id}/history. Every nested deep agent reported an empty messages channel while its tasks had demonstrably run many model turns and tool calls — with 1452 kB of message blobs sitting in checkpoint_blobs for one such namespace. Plain agents, whose messages is a BinaryOperatorAggregate rather than a DeltaChannel, were unaffected.

System info

langgraph main @ 1.2.10 (also reproduced on 1.2.1), Python 3.13, macOS.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions