Skip to content

Commit d89c8f8

Browse files
Cristhianzlautofix-ci[bot]erichare
authored
fix(hitl): rename inputs, advanced timeout + HITL palette and control polish (#14090)
* fix(hitl): rename inputs, advanced timeout (backend) + fix(frontend): HITL palette and control polish * [autofix.ci] apply automated fixes * fix(ci): refresh generated assets * fix(hitl): atomic supersede + close stale cards * remove unecessary files * [autofix.ci] apply automated fixes * add hitl metadata * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Hare <ericrhare@gmail.com>
1 parent 82876e4 commit d89c8f8

38 files changed

Lines changed: 909 additions & 249 deletions

File tree

.secrets.baseline

Lines changed: 63 additions & 63 deletions
Large diffs are not rendered by default.

docs/features/durable-execution-hitl.md

Lines changed: 26 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,18 @@ The pause probe is a no-op unless a component requests it, so normal flows are b
127127
- **Then** the run finalizes once — no loop back to the first node
128128
- **And** only the chosen branch of each node runs (the first node's non-chosen branches stay dead across the second pause's resume, not just within the build that answered them)
129129

130+
### Scenario: Rerunning a paused flow supersedes the stale pause
131+
- **Given** a flow with a `SUSPENDED` run awaiting a decision
132+
- **When** the same user submits a new background run of that flow
133+
- **Then** the stale suspended run is cancelled (`CANCELLED`, pending request cleared) before the new run starts, so only the new pause is offered on every surface
134+
- **And** the persisted chat card of the superseded pause is stamped `superseded` — after a history reload it renders closed ("Superseded by a new run") instead of turning interactive again and 409ing on every click
135+
- **And** the scope is flow + user on the langflow backend (another user's pause on the same flow is untouched) and flow on `lfx serve` (single API-key identity); running jobs are never superseded, so parallel runs stay supported
136+
137+
### Scenario: Supersede loses the race to a resume
138+
- **Given** a `SUSPENDED` run whose resume arrives while a rerun's supersede is in flight
139+
- **When** the resume wins the atomic `SUSPENDED → IN_PROGRESS` claim between supersede's select and its cancel
140+
- **Then** supersede skips that job entirely — no `CANCELLED` write, no checkpoint delete, no metadata clear (backend) and no lingering STOP signal (`lfx serve`) — and the resumed run finishes normally
141+
130142
### Scenario: Single-flight resume
131143
- **Given** two resume requests for the same `SUSPENDED` job
132144
- **When** both arrive
@@ -225,12 +237,19 @@ Drain the queue **inline** after cancelling the worker (`service.py::_stop`), an
225237

226238
## 6. Technical Specification
227239

228-
### 6.1 Pause → suspend (initial run)
240+
### 6.1 Submit supersedes stale pauses
241+
- `BackgroundExecutionService.submit` calls `supersede_suspended_runs(flow_id, user_id)` after `create_job` (so idempotent retries still return the existing job) and before enqueue: every `SUSPENDED` workflow job of the same flow + user is cancelled through the `_cancel_suspended` path (status `CANCELLED`, card stamped superseded, checkpoint deleted, pending cleared, `run_cancelled` event, bus closed).
242+
- The cancel is an **atomic claim**: `JobService.claim_suspended_for_cancel` (a conditional `UPDATE … WHERE status = SUSPENDED`, the mirror of `claim_suspended_for_resume`) flips the row, and cleanup runs only for rows this caller actually claimed. A resume that won the flip first keeps its run — supersede can never cancel a resumed job or destroy its checkpoint mid-flight. `stop_job` uses the same claim and falls through to the running-job STOP path when it loses.
243+
- Before metadata is cleared, `mark_card_superseded` (`api/v2/hitl.py`) patches the persisted card's `human_input` content with `superseded: true` (skipping already-answered cards); `HumanInputCard` renders that state closed, so a reloaded history cannot re-offer a decision that would only 409.
244+
- `DurableServeWorkflowHost.submit_background` mirrors it flow-scoped: `SqliteDurableJobStore.claim_suspended_for_cancel` first, then (only for claimed rows) STOP signal, task cancel, pending metadata cleared — a lost claim also means no stray STOP signal for the resumed continuation to consume.
245+
- Frontend: the canvas badge auto-open is keyed by `request_id`, so the superseding run's new pause reopens a dismissed popover; the 5s pending poll converges every surface onto the single remaining pause.
246+
247+
### 6.2 Pause → suspend (initial run)
229248
- The graph consults a pause probe at each layer boundary; on a pending pause it snapshots itself, writes a `checkpoint` row (always-writable schema), and raises `GraphPausedException`.
230249
- The build seam (`api/build.py`) catches it, persists the human-input card to history, emits `human_input_required`, and ends the stream without `on_end`.
231250
- The runner marks the job `SUSPENDED` (`services/background_execution/runner.py`).
232251

233-
### 6.2 Resume (single-flight)
252+
### 6.3 Resume (single-flight)
234253
- `POST /api/v2/workflows/{job_id}/resume` with `{request_id, decision}`.
235254
- The route re-enforces `FlowAction.EXECUTE` (`ensure_resume_execute_permission`) before applying the decision — job ownership alone is not enough once shared access has been revoked.
236255
- `claim_suspended_for_resume` performs the atomic status claim; failure → `NOT_RESUMABLE`.
@@ -253,7 +272,7 @@ Drain the queue **inline** after cancelling the worker (`service.py::_stop`), an
253272
# restore built vertices; re-run only non-input predecessors of the gated vertex
254273
```
255274

256-
### 6.3 Trace continuity (initial run)
275+
### 6.4 Trace continuity (initial run)
257276
- `build_graph_from_data` (`api/utils/flow_utils.py`) pins the run id before tracing starts:
258277
```python
259278
if (caller_run_id := kwargs.get("run_id")) is not None:
@@ -262,15 +281,15 @@ Drain the queue **inline** after cancelling the worker (`service.py::_stop`), an
262281
```
263282
- `create_graph` forwards `run_id=str(job_id) if job_id is not None else run_id` to both `build_graph_from_data` and `build_graph_from_db`.
264283

265-
### 6.4 Gate span recording
284+
### 6.5 Gate span recording
266285
- `TracingService.record_event_span(span_id, name, outputs, span_type="chain")` writes a standalone start+end span into the active trace for every ready tracer; deactivated/contextless calls are no-ops.
267286

268-
### 6.5 Frontend (reads backend spans)
287+
### 6.6 Frontend (reads backend spans)
269288
- `useGetTraceQuery``GET /api/v1/monitor/traces/{id}` returns the span tree.
270289
- `TraceDetailView.tsx`: renders the backend gate span; `backendHasGate` dedup prevents a duplicate synthetic node; `executedOutputSpans` bridges Chat Output from live `flowPool` only during the resume window (deduped by name).
271290
- `hitlStore.ts`: in-memory only (`pending` slot); `localStorage`/`persist` removed.
272291

273-
### 6.6 lfx serve durable mode (LE-1695)
292+
### 6.7 lfx serve durable mode (LE-1695)
274293

275294
Bare `lfx serve` gains the same background + HITL contract without a database, backed by the single-node SQLite substrate:
276295

@@ -282,7 +301,7 @@ Bare `lfx serve` gains the same background + HITL contract without a database, b
282301
- **Opt-in**: `LFX_SERVE_DURABLE_DB=<path>`. Unset → serve stays stateless and `mode: background` keeps its 422; workers in multi-worker mode inherit the env var and share the DB file (a stop cancels the run only in the worker executing it).
283302
- **Proof**: `src/lfx/tests/unit/cli/test_serve_durable.py` — background echo completes with the sync-shaped response; a HITL flow suspends, exposes its pending request, resumes only the approved branch, and resumes to completion on a fresh app instance over the same DB file (restart equivalence). The `{job_id}/events` SSE stream replays a completed run, surfaces the `human_input_request` on a suspend, and — reconnected with `Last-Event-ID` after a resume — delivers only the post-resume `human_input_decision` + `end` frames.
284303

285-
### 6.7 Component map (quick index)
304+
### 6.8 Component map (quick index)
286305

287306
| Concern | Where | Key symbols |
288307
|---------|-------|-------------|

src/backend/base/langflow/api/v2/hitl.py

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,6 +166,54 @@ def _set_submitted_action(content_blocks: list, action_id: str | None, request_i
166166
return changed
167167

168168

169+
def _mark_superseded(content_blocks: list) -> bool:
170+
"""Flag every still-open human_input content as superseded, in place."""
171+
changed = False
172+
for block in content_blocks or []:
173+
contents = block.get("contents") if isinstance(block, dict) else getattr(block, "contents", None)
174+
for content in contents or []:
175+
is_dict = isinstance(content, dict)
176+
ctype = content.get("type") if is_dict else getattr(content, "type", None)
177+
if ctype != "human_input":
178+
continue
179+
answered = content.get("submitted_action") if is_dict else getattr(content, "submitted_action", None)
180+
if answered is not None:
181+
continue
182+
if is_dict:
183+
content["superseded"] = True
184+
else:
185+
content.superseded = True
186+
changed = True
187+
return changed
188+
189+
190+
async def mark_card_superseded(job_id: UUID) -> None:
191+
"""Close the persisted card of a run cancelled by supersede/stop.
192+
193+
Without this the reloaded card (no ``submitted_action``) turns interactive
194+
again and every click 409s against the cancelled job. Patched in place like
195+
``mark_card_answered`` so prompt/options are preserved for the history.
196+
"""
197+
from lfx.services.deps import session_scope
198+
199+
from langflow.services.database.models.message.model import MessageTable
200+
201+
job = await get_job_service().get_job_by_job_id(job_id)
202+
card_message_id = (job.job_metadata or {}).get("card_message_id") if job else None
203+
if not card_message_id:
204+
return
205+
try:
206+
async with session_scope() as session:
207+
message = await session.get(MessageTable, UUID(str(card_message_id)))
208+
if message is None:
209+
return
210+
if _mark_superseded(message.content_blocks):
211+
flag_modified(message, "content_blocks")
212+
session.add(message)
213+
except Exception as _e: # noqa: BLE001
214+
await logger.awarning("Failed to mark human-input card superseded for job %s: %r", job_id, _e)
215+
216+
169217
async def mark_card_answered(job_id: UUID, request_id: str, decision: dict, *, card_message_id=None) -> None:
170218
"""Record the chosen action on the persisted card message for ``job_id``.
171219

src/backend/base/langflow/initial_setup/starter_projects/Document Q&A.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -700,7 +700,7 @@
700700
},
701701
{
702702
"name": "cryptography",
703-
"version": "48.0.1"
703+
"version": "49.0.0"
704704
},
705705
{
706706
"name": "langchain_chroma",

src/backend/base/langflow/initial_setup/starter_projects/Knowledge Retrieval.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -706,7 +706,7 @@
706706
},
707707
{
708708
"name": "cryptography",
709-
"version": "48.0.1"
709+
"version": "49.0.0"
710710
},
711711
{
712712
"name": "langchain_chroma",

src/backend/base/langflow/initial_setup/starter_projects/Vector Store RAG.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1169,7 +1169,7 @@
11691169
},
11701170
{
11711171
"name": "cryptography",
1172-
"version": "48.0.1"
1172+
"version": "49.0.0"
11731173
},
11741174
{
11751175
"name": "langchain_chroma",

src/backend/base/langflow/locales/en.json

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3925,13 +3925,13 @@
39253925
"components.huggingfacemodel.inputs.typical_p.info.ed0e56c7": "Typical Decoding mass.",
39263926
"components.huggingfacemodel.outputs.model_output.display_name.698eabeb": "Language Model",
39273927
"components.huggingfacemodel.outputs.text_output.display_name.2544139a": "Model Response",
3928-
"components.humaninput.description.e35d40a8": "Pause the flow to collect a human decision and route on the chosen action.",
3928+
"components.humaninput.description.548359b3": "Pause the flow for a human-in-the-loop (HITL) decision and route on the chosen action.",
39293929
"components.humaninput.display_name.d7c8a913": "Human Input",
3930-
"components.humaninput.inputs.decisions.display_name.15670189": "User Actions",
3931-
"components.humaninput.inputs.decisions.info.5126a1fd": "Actions the human can choose; each becomes a branch output. Pick from the list or type a custom one.",
3930+
"components.humaninput.inputs.decisions.display_name.8ce3a776": "User Choices",
3931+
"components.humaninput.inputs.decisions.info.ad649edf": "Choices the human can pick; each becomes a branch output. Pick from the list or type a custom one.",
39323932
"components.humaninput.inputs.enable_fallback.display_name.109c8318": "Enable Fallback",
39333933
"components.humaninput.inputs.enable_fallback.info.72174e80": "Add a 'fallback' output taken when the answer arrives after the timeout window.",
3934-
"components.humaninput.inputs.prompt.display_name.6a8db532": "Form Content",
3934+
"components.humaninput.inputs.prompt.display_name.36ecb4f8": "Input",
39353935
"components.humaninput.inputs.prompt.info.e40f2bd4": "Content shown to the human for review.",
39363936
"components.humaninput.inputs.timeout.display_name.70594d93": "Timeout",
39373937
"components.humaninput.inputs.timeout.info.866d9f42": "A response received after this window is rerouted to the fallback path (when enabled) instead of the chosen action. The run stays paused until a response arrives or the server's suspended-run deadline expires it. Set to 0 to wait indefinitely.",

src/backend/base/langflow/services/background_execution/service.py

Lines changed: 52 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -162,6 +162,39 @@ async def teardown(self) -> None:
162162

163163
# ------------------------------------------------------------------ submit
164164

165+
async def supersede_suspended_runs(self, *, flow_id: UUID, user_id: UUID) -> list[UUID]:
166+
"""Cancel this user's SUSPENDED runs of the flow so a rerun replaces the stale pause.
167+
168+
A suspended run holds a pause nobody will answer once its owner reruns the flow;
169+
left alive it piles up in every pending surface (badge, cards, trace bar). Scope is
170+
flow + user: another user's pause on the same flow is never touched, and running
171+
jobs are untouched so parallel runs stay supported.
172+
"""
173+
from sqlmodel import select
174+
175+
from langflow.services.database.models.jobs.model import Job
176+
from langflow.services.deps import session_scope
177+
178+
job_service = get_job_service()
179+
async with session_scope() as session:
180+
result = await session.exec(
181+
select(Job.job_id).where(
182+
Job.flow_id == flow_id,
183+
Job.user_id == user_id,
184+
Job.status == JobStatus.SUSPENDED,
185+
Job.type == JobType.WORKFLOW,
186+
)
187+
)
188+
stale_job_ids = list(result.all())
189+
cancelled: list[UUID] = []
190+
for stale_job_id in stale_job_ids:
191+
if self._scaled:
192+
await self._backend.stop(str(stale_job_id))
193+
cancelled.append(stale_job_id)
194+
elif await self._cancel_suspended(stale_job_id, job_service):
195+
cancelled.append(stale_job_id)
196+
return cancelled
197+
165198
async def submit(self, *, flow_id: UUID, request: dict[str, Any], user: UserRead) -> UUID:
166199
# Lazy-start the executor so the facade works whether or not the app
167200
# lifespan called start() first. start() is idempotent.
@@ -199,6 +232,9 @@ async def submit(self, *, flow_id: UUID, request: dict[str, Any], user: UserRead
199232
# variables by name for background runs rather than passing secrets inline.
200233
# The live in-memory run below still uses the full ``request``.
201234
await job_service.update_job_metadata(job_id, {"request": self._redact_request(request)})
235+
# After create_job so an idempotent retry returns the existing job instead of
236+
# cancelling it; the new job is QUEUED, so the suspended-only query skips it.
237+
await self.supersede_suspended_runs(flow_id=flow_id, user_id=user.id)
202238
if self._scaled:
203239
# Scaled mode: hand the QUEUED job id to a worker via the redis claim
204240
# queue; the worker hydrates the request from the job row.
@@ -366,8 +402,9 @@ async def stop_job(self, job_id: UUID, user: UserRead) -> None:
366402
await self._backend.stop(str(job_id))
367403
return
368404
job_service = get_job_service()
369-
if job.status == JobStatus.SUSPENDED:
370-
await self._cancel_suspended(job_id, job_service)
405+
# A lost claim means a resume flipped it IN_PROGRESS mid-call — fall
406+
# through to the running-job stop path instead of doing nothing.
407+
if job.status == JobStatus.SUSPENDED and await self._cancel_suspended(job_id, job_service):
371408
return
372409
await job_service.write_signal(job_id, SignalType.STOP)
373410
await self._executor.cancel(str(job_id))
@@ -408,14 +445,25 @@ async def resume_job(self, job_id: UUID, user: UserRead, *, request_id: str, dec
408445
raise
409446
return True
410447

411-
async def _cancel_suspended(self, job_id: UUID, job_service) -> None:
412-
await job_service.update_job_status(job_id, JobStatus.CANCELLED, finished_timestamp=True)
448+
async def _cancel_suspended(self, job_id: UUID, job_service) -> bool:
449+
"""Cancel a SUSPENDED job; False when a concurrent resume already claimed it.
450+
451+
The conditional claim is what makes supersede-vs-resume safe: cleanup
452+
(checkpoint delete, metadata clear, terminal event) runs only for the row
453+
this caller actually flipped, never for a run a resume just revived.
454+
"""
455+
if not await job_service.claim_suspended_for_cancel(job_id):
456+
return False
457+
from langflow.api.v2.hitl import mark_card_superseded
458+
459+
await mark_card_superseded(job_id)
413460
with contextlib.suppress(Exception):
414461
await job_service.delete_checkpoint(job_id, "graph")
415462
await job_service.update_job_metadata(job_id, {"pending_request_id": None})
416463
await job_service.append_event(job_id, "run_cancelled", {"type": "cancelled"})
417464
await job_service.consume_signals(job_id, SignalType.STOP)
418465
await self._bus.close(str(job_id))
466+
return True
419467

420468
# ----------------------------------------------------------- startup sweep
421469

src/backend/base/langflow/services/jobs/service.py

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -615,6 +615,26 @@ async def claim_suspended_for_resume(self, job_id: UUID, *, owner: str | None =
615615
await session.flush()
616616
return result.rowcount == 1
617617

618+
async def claim_suspended_for_cancel(self, job_id: UUID) -> bool:
619+
"""Atomically flip SUSPENDED->CANCELLED; True iff this caller won.
620+
621+
Mirror of ``claim_suspended_for_resume`` for the supersede/stop side of the
622+
race: a resume that already flipped the row to IN_PROGRESS keeps its run —
623+
an unconditional CANCELLED write here would clobber it and destroy its
624+
checkpoint mid-flight.
625+
"""
626+
from sqlmodel import update
627+
628+
async with session_scope() as session:
629+
stmt = (
630+
update(Job)
631+
.where(Job.job_id == job_id, Job.status == JobStatus.SUSPENDED)
632+
.values(status=JobStatus.CANCELLED, finished_timestamp=datetime.now(timezone.utc))
633+
)
634+
result = await session.exec(stmt) # type: ignore[call-overload]
635+
await session.flush()
636+
return result.rowcount == 1
637+
618638
async def queued_workflow_job_ids(self) -> list[UUID]:
619639
"""Return the ids of every QUEUED workflow job (for strand recovery)."""
620640
async with session_scope() as session:

0 commit comments

Comments
 (0)