fix(graph): rollback from incomplete checkpoint messages#4791
Open
Jupiter363 wants to merge 1 commit into
Open
fix(graph): rollback from incomplete checkpoint messages#4791Jupiter363 wants to merge 1 commit into
Jupiter363 wants to merge 1 commit into
Conversation
Signed-off-by: Jupiter363 <1404785625@qq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe what this PR does / why we need it
Fixes #4789.
When a graph stream is cancelled or times out after a tool node has already persisted
ToolResponseMessage, the following streaming model node may never persist the finalAssistantMessage. The next user turn can then load an incomplete checkpoint and build an invalid message sequence such as:That sequence is not accepted by OpenAI/DashScope-compatible chat APIs and can make the next model request fail.
This PR replaces #4790 with a smaller rollback-oriented fix. The previous background-continuation approach tried to keep the cancelled run alive internally, but that introduced extra concurrency risks between an old background run and a newer turn on the same thread. This PR keeps normal Reactor cancellation semantics and instead makes the next turn recover from the latest safe checkpoint.
Does this pull request fix one issue?
Fixes #4789
Describe how you did it
This PR adds a checkpoint message-sequence validator and uses it when
CompiledGraph.getInitialState(...)loads checkpoint state for a new user input.If the latest checkpoint contains an incomplete tool-call sequence, the graph skips it and rolls back to the nearest earlier checkpoint whose
messageshistory can safely accept another user message.The validator rejects persisted histories such as:
tool -> usercontinuation.It accepts histories where the tool-call chain has been closed by a final assistant message.
This PR does not rewrite checkpoint records, delete checkpoint records, fabricate a placeholder assistant message, or keep cancelled graph executions running in the background. It only changes which checkpoint is selected as the base state for the next user input.
Why rollback instead of background continuation
A background-continuation approach can preserve the cancelled turn's final assistant output when it eventually completes, but it also creates a race: the old cancelled run may keep appending checkpoints while the user immediately sends the next message on the same thread. That requires additional lineage guards around checkpoint writes and
releaseThread(true), and still leaves backend-level CAS/distributed-locking concerns for multi-instance deployments.The rollback approach is more conservative for this issue:
Describe how to verify it
I verified this PR with:
mvn -pl spring-ai-alibaba-graph-core -Dtest=CheckpointRollbackConsistencyTest,MessageSequenceValidatorTest,GraphRunnerContextResumeMessageOrderTest,StreamFunctionCallTest test mvn -pl spring-ai-alibaba-graph-core checkstyle:check mvn -pl spring-ai-alibaba-graph-core spotless:check git diff --checkLatest local result:
Regression coverage includes:
ToolResponseMessagerolls back to the previous safe checkpoint;toolhistories without a final assistant;assistant(tool_calls) -> tool -> assistantchain;