Skip to content

DeltaChannel: forking a thread replays the abandoned branch's writes into the fork #8443

Description

@AnnaSuSu

Checked other resources

  • I added a clear and descriptive title that summarizes this issue.
  • I used the GitHub search to find a similar question and didn't find it.
  • I am sure that this is a bug in LangGraph rather than my code.
  • The bug is not resolved by updating to the latest stable version of LangGraph.
  • This is not related to the langchain-community package.
  • I posted a self-contained, minimal, reproducible example. A maintainer can copy it and run it AS IS.

Related Issues / PRs

#8382 and #8384 are also DeltaChannel bugs but unrelated to this one (write ordering within a superstep, and the InMemorySaver migration override).

Reproduction Steps / Example Code (Python)

import asyncio
from typing import Annotated, TypedDict

from langgraph.channels.delta import DeltaChannel
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.graph import StateGraph


def append(current, writes):
    out = list(current or [])
    for w in writes:
        out.extend(w if isinstance(w, list) else [w])
    return out


class S(TypedDict):
    log: Annotated[list, DeltaChannel(append, snapshot_frequency=1000)]


def build(cp, tag):
    async def node(state):
        return {"log": [f"{tag}-out"]}

    b = StateGraph(S)
    b.add_node("n", node)
    b.set_entry_point("n")
    b.set_finish_point("n")
    return b.compile(checkpointer=cp)


async def main():
    cp = InMemorySaver()
    cfg = {"configurable": {"thread_id": "t"}}
    await build(cp, "first").ainvoke({"log": ["in-1"]}, cfg)
    g = build(cp, "second")
    await g.ainvoke({"log": ["in-2"]}, cfg)

    # the checkpoint before "in-2" entered state
    base = None
    async for snap in g.aget_state_history(cfg):
        if "in-2" not in snap.values["log"]:
            base = snap
            break
    print("fork base:", base.values["log"])

    fork_cfg = {
        "configurable": {
            "thread_id": "t",
            "checkpoint_ns": "",
            "checkpoint_id": base.config["configurable"]["checkpoint_id"],
        }
    }
    await build(cp, "third").ainvoke({"log": ["in-3"]}, fork_cfg)
    print("actual  :", (await g.aget_state(cfg)).values["log"])
    print("expected:", base.values["log"] + ["in-3", "third-out"])


asyncio.run(main())

Output:

fork base: ['in-1', 'first-out']
actual  : ['in-1', 'first-out', 'in-2', 'in-3', 'third-out']
expected: ['in-1', 'first-out', 'in-3', 'third-out']

'in-2' belongs to the branch that was abandoned by forking, and it should not be in the fork's state.

update_state against an older checkpoint has the same problem — replacing the last two lines with

    forked = await g.aupdate_state(fork_cfg, {"log": ["patched"]})
    print((await g.aget_state(forked)).values["log"])

gives ['in-1', 'x-out', 'patched', 'in-2'] instead of ['in-1', 'x-out', 'patched'].

Description

Addressing an older checkpoint creates a fork: the shared parent ends up with two children. get_delta_channel_history walks the parent chain and collects every pending_writes entry it finds on each on-path ancestor — but the shared parent also holds the writes of the sibling branch, and nothing in the stored data records which child consumed which write. So the abandoned branch's writes are replayed into the fork.

The method's docstring says:

Walks the parent chain (not list(before=...)): for forked threads, only on-path ancestors contribute.

The ancestors it selects are indeed on-path. The writes hanging off them are not.

This is not specific to one saver: InMemorySaver, the base walk (used by SQLite), and PostgresSaver's own paged-SQL override all reproduce it, which suggests write-to-child ownership is a gap in the delta contract rather than one implementation's slip. full (non-delta) channels are unaffected, since their checkpoints carry complete channel_values and need no replay.

I hit this in a chat application where "regenerate the last answer" resumes from the checkpoint before that turn: the superseded answer came back alongside the new one after a reload.

Proposed fix — write ownership is not recoverable at read time, so make the fork's first checkpoint self-contained instead: force delta channels to snapshot when a run is entered against an explicitly addressed checkpoint, and when update_state writes against one. The walk then terminates at the fork instead of reaching the shared parent. This mirrors the existing force-snapshot for Overwrite writes, which exists for the same reason. Cost is one extra snapshot per explicitly addressed resume — not per superstep.

I have this implemented and tested against main (_loop.py, _checkpoint.py, main.py, plus a regression test file; make format / make lint clean, and the full libs/langgraph suite shows no new failures versus unpatched main). Happy to open the PR — could a maintainer assign me to this issue?

System Info

langgraph 1.2.9 (and reproduced on main @ 30c4d58), langgraph-checkpoint 4.1.1, langgraph-checkpoint-sqlite 3.1.0
Python 3.11.5, macOS 26.4 arm64

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions