feat(api): durable background execution service (store + default backend) - #13507
Merged
Conversation
Rebased onto release-1.10.0. The base independently rebuilt the v2 workflows backend (RBAC, body globals, share-aware fetch); keep our forward design and conform its auth to that work: 1. Auth: keep get_current_user_for_workflow (session-or-API-key authN that does not hold a DB connection during the inline run, avoiding the SQLite lock contention api_key_security would cause) and enforce the base's RBAC on top: ensure_flow_permission(EXECUTE) before run, (READ) before status reconstruct, with widen_for_shares fetch. 2. Port the base's request-body globals onto the v2 WorkflowRunRequest. The X-LANGFLOW-GLOBAL-VAR-* headers stay supported (the Responses API passes globals that way); body globals win on conflict. Converters echo the effective globals via effective_globals. 3. Public endpoint keeps the v1 build_public_tmp posture (access_type==PUBLIC, run-as-owner); RBAC applies to the authenticated endpoint only. 4. Preserve the base's post-build KB-cache invalidation in the AG-UI build path. The endpoint, AG-UI bridge, pluggable stream adapters, public endpoint, and re-attach are unchanged.
The synchronous /api/v2/workflows response keyed every result under its component id, so reading the answer meant knowing an id you can't predict. Surface two additive fields: - output_text: the flow's single text answer (ChatOutput/TextOutput). None when the flow has zero or multiple text outputs, so callers read outputs rather than the shortcut guessing which channel is the answer. - session_id: echoes the resolved session so chat/memory callers can continue the same thread (v1 /run returned this; v2 had dropped it). outputs is unchanged, so this is non-breaking.
…ponse
Pin the sync-response shortcuts on the v2 workflows endpoint:
- output_text surfaces the lone ChatOutput/TextOutput text and stays None for
non-output message nodes, data-only flows, and multi-text flows
- session_id echoes the resolved session; the error response exposes neither
- each outputs entry exposes only {type, status, content, metadata}, with the
component id carried by the dict key
Also drop the component_id kwarg the converter passed to ComponentOutput, which
has no such field and silently dropped it.
Replace the flat output_text shortcut with an `output` object carrying the resolved text answer plus a `reason` that explains why it resolved that way (single/multiple/none/non_string/failed), so a null answer is always diagnosable instead of silently None. `reason` follows the LLM-domain finish_reason/stop_reason convention, distinct from the lifecycle status. Also add `display_name` to each ComponentOutput (the stable component id stays the dict key) and a computed `has_errors` flag derived from errors.
Let a sync caller name the output(s) they want via output_ids so output.text resolves deterministically (reason=single) on multi-output flows instead of going null. Selection is steer-only: it picks the answer among the named outputs without filtering the outputs map. Invalid ids are rejected with 422 before the flow runs (and before any job row is created), so a typo costs no compute. Resolution considers selected outputs that actually fired, so branching flows resolve to whichever candidate ran.
Give v2-workflows sync and the langflow stream protocol one parser. The stream now emits a normalized "output" event per terminal output carrying an OutputEvent (the ComponentOutput shape sync returns in outputs[id], plus component_id). A shared build_component_output() backs both the sync converter and the adapter, and the build loop ships authoritative vertex metadata as an additive output_meta key on end_vertex (existing consumers read build_data and ignore it). This is access-pattern parity (one parser, same fields, same terminal set), not byte-identical content: the stream reuses the v1 build path whose display serialization differs from sync's run_graph output.
…ackend) Turns v2 mode:background into a durable, in-API background execution service behind a BackgroundExecutionService facade. Adds the store layer (result/error columns, job_events durable milestone log, execution_signals control, heartbeat/lease, 3 migrations), the default backend (bounded executor, runner, in-memory live bus, liveness-aware single-flight orphan sweep), the v2 endpoint rewiring, and the real-instance test harness. Needs no new infra; works on the SQLite single-process install. The redis-scaled worker backend is stacked on top in a follow-up PR.
Contributor
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
WalkthroughIntroduces a durable background job execution system: three Alembic migrations add ChangesDurable Background Execution + AG-UI Streaming v2
Sequence Diagram(s)sequenceDiagram
rect rgba(70, 130, 180, 0.5)
note over FlowStore,Server: Browser Build Flow (AG-UI path)
end
participant FlowStore as flowStore.buildFlow
participant Bridge as runFlowAGUI
participant Server as POST /api/v2/workflows
participant BES as BackgroundExecutionService
participant Runner as JobRunner
participant JobSvc as JobService
FlowStore->>Bridge: runFlowAGUI({flowId, mode="stream", signal})
Bridge->>Server: POST (stream_protocol="agui", SSE)
Server->>Server: parse WorkflowRunRequest → ParsedWorkflowRun
Server->>Server: _stream_event_frames(adapter=AGUIAdapter)
loop SSE frames
Runner->>JobSvc: append_event(durable_seq)
Runner-->>Server: (frame_bytes, event_type)
Server-->>Bridge: SSE: RUN_STARTED / STATE_DELTA / RUN_FINISHED
Bridge->>Bridge: handleAGUIEvent → applyStateDelta
Bridge->>Bridge: update flowStore / messagesStore
end
Bridge-->>FlowStore: resolve (success/error/abort)
FlowStore->>FlowStore: trackFlowBuild(buildInfo)
sequenceDiagram
rect rgba(60, 179, 113, 0.5)
note over Client,JobSvc: Background job submission and re-attach
end
participant Client as HTTP Client
participant API as workflow.py
participant BES as BackgroundExecutionService
participant JobSvc as JobService
participant Executor as InProcessExecutor
participant Bus as InMemoryLiveBus
Client->>API: POST /workflows (mode=background)
API->>BES: submit(flow_id, request, user)
BES->>JobSvc: create_job(dedupe_key, user_id)
BES->>Executor: submit(job_id, coro_factory)
API-->>Client: WorkflowJobResponse {job_id, links.events}
note over Executor,Bus: Job runs asynchronously
Executor->>Bus: publish(job_id, LiveFrame)
Executor->>JobSvc: append_event / set_result
Client->>API: GET /workflows/{job_id}/events (Last-Event-ID)
API->>BES: events(job_id, last_event_id, user)
BES->>JobSvc: read_events(after_seq=last_seq)
BES->>Bus: reattach(job_id, last_seq, read_durable)
Bus-->>Client: replayed durable frames + live tail SSE
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 8 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (8 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Author
This was referenced Jun 4, 2026
Contributor
|
✅ Migration Validation Passed All migrations follow the Expand-Contract pattern correctly. |
dkaushik94
reviewed
Jun 5, 2026
The hard_proof marker name was a vibe word that said nothing about what the tests need. Rename it to real_services everywhere: the pytest marker registration, the *_hard_proof.py test files, the Makefile target (real_services_tests), the -m selector in migration-validation.yml, and the CI job. real_services says what these tests require: real Postgres + Redis + worker subprocesses. (integration was already taken for the external-API suite under tests/integration.)
github-actions
Bot
added
enhancement
dkaushik94
reviewed
Jun 8, 2026
…kflows-agui # Conflicts: # src/frontend/src/stores/flowStore.ts
The v2 public endpoint only ran validate_flow_for_current_settings and skipped validate_public_flow_no_code_execution, which the v1 build_public_tmp path applies. A public flow containing a Python interpreter/REPL (or the legacy Python Code Structured tool, Smart Transform lambda) was therefore an unauthenticated server-side code-execution primitive (report H1-3754930). Mirror v1: import the validator and call it right after the public-access gate. PublicFlowValidationError subclasses CustomComponentValidationError, so the existing handler already sanitizes it to a 400 'This flow cannot be executed.' without leaking the blocked component class names. Add a non-mocking test that builds a public flow with a real PythonREPLComponent and asserts the sanitized 400 (verified RED: returns 200 without the gate). LE-1389
…rtex builds A completed background job's GET status 500'd with 'No vertex builds found for job_id'. The background build path differed from the sync path twice: 1. generate_flow_events minted a fresh run_id instead of using job_id, so vertex builds were keyed by an id the status query never uses. Thread run_id through _stream_event_frames -> generate_flow_events and pass job_id from the background buffer so graph.run_id == job_id (the sync path already does graph.set_run_id(job_id)). 2. The SSE build loop (build_vertices) only persisted builds when log_builds was set and never passed job_id. Tie log_builds to job-tracked runs (run_id present) and pass job_id=graph.run_id on the persist call. Job-tracked runs also persist streaming terminal vertices so reconstruction is complete; the live build path (run_id is None) keeps its original behavior, so the v1 build path is unchanged. Test: a real background run polled to completion, then GET status asserts a reconstructed 200 (verified RED: 500 'No vertex builds found' before the fix). Covers the non-streaming flow. v1 build path unchanged (35 build tests pass); AG-UI suite 46 pass. LE-1389
dkaushik94
reviewed
Jun 9, 2026
Comment on lines
+960
to
+977
| # Reconstruct response from vertex_build table (sync path persists | ||
| # those keyed by job_id). Background runs do not write vertex_builds | ||
| # keyed by job_id, so reconstruction finds nothing and raises | ||
| # ValueError — fall back to the durable Job.result the runner wrote | ||
| # so a completed background run reports completed instead of 500ing. | ||
| try: | ||
| return await reconstruct_workflow_response_from_job_id( | ||
| session=session, | ||
| flow=flow, | ||
| job_id=job_id_str, | ||
| user_id=str(current_user.id), | ||
| ) | ||
| except ValueError: | ||
| return WorkflowExecutionResponse( | ||
| flow_id=flow_id_str, | ||
| job_id=job_id_str, | ||
| status=JobStatus.COMPLETED, | ||
| ) |
Member
There was a problem hiding this comment.
@ogabrielluiz
What are we planning with the persisting final workflow outputs? Looks like we have the durable events that can be replayed to construct the final response and/or end_vertex events that can be marked as output nodes to get their outputs as the final results.
Should we store the terminal vertex outputs in a workflow response table or under jobs.result that the sync path and the async path both write to, to ensure a single place of results? Sync/Async is the method of execution, but the expectation of outcomes would always remain the same, right? Thoughts?
I suggest we write to to job.result the final outputs so that it proves to be the source of truth for any runs that have been completed. Do you think a dedicated WorkflowResults table is a better way considering jobs can be more than just workflows (Ingestion, Memory LLM Preprocessing etc which will lead to different shaped outputs save to the job.results JSONB format)
Contributor
Author
There was a problem hiding this comment.
Hey @dkaushik94, good question. Right now the two paths don't share one results home, and I think they should. The sync path reconstructs the response from vertex_build rows keyed by job_id, and the background path falls back to the durable Job.result the runner persisted (workflow.py around the reconstruct_workflow_response_from_job_id -> ValueError fallback). So the result is already in Job.result for background runs, just not for sync.
I lean toward Job.result as the single source of truth rather than a dedicated WorkflowResults table, for the reason you gave: jobs are more than workflows (ingestion, memory preprocessing), so a workflow-specific table would force a second results home the moment we add a non-workflow job. The JSONB result column already absorbs differently-shaped outputs.
The piece I'd want to nail down is the shape we write, so the reconstruction code can stop being path-specific and just read Job.result. I think that's worth a follow-up rather than this PR, since making sync write Job.result too touches the sync orchestration. What do you think, keep Job.result as the contract and converge both paths onto it next?
Member
There was a problem hiding this comment.
Yes, I think we should converge to Job.results as the canonical result field for all jobs. Let's do a follow-up on this to have sync and async both converge. Thanks @ogabrielluiz 👍🏼
Do you want to track this somehow? I can take up the follow up as well if you need me to. Lmk.
dkaushik94
reviewed
Jun 9, 2026
| @@ -85,11 +85,21 @@ | |||
| from langflow.services.database.models.flow.model import FlowRead | |||
Member
There was a problem hiding this comment.
This is probably not worth doing in this PR, but I want to surface the pattern of instrumentation rather than the scope this PR is addressing, which is the infra for handling what is generated.
We currently use generate_flow_events() and have multiple call sites to emit dedicated events. A more maintainable and enforced way would be the ABC design.
Follow-up PR suggestion: Observable base & @observable decorator
(Suggesting as a separate PR on top of this one — not asking for changes here.)
Right now, every place we want to emit a workflow event has to thread event_manager through and call event_manager.on_something(...) by hand. That's fine when there are a few call sites, but it scales badly: each new event lives at the call site, custom component authors can silently emit nothing, and there's no contract that says "an observable thing produces these events."
I'd like us to explore moving to a pattern where any class that participates in execution — Graph, Vertex, Component, the LangChain callback bridge inherits from a small Observable base that requires three callbacks: one for when work starts, one for when work finishes, one for when it errors (and this is felxible we can have more methods for special cases). A decorator on each runnable method then handles the actual publication, reading the current run's event manager from context. Authors stop calling event_manager directly; they describe what their event looks like, and the decorator decides when to fire it.
The payoff is the kind that doesn't show up in a single diff but compounds across the codebase. New components get instrumented the moment they inherit. Forgetting to wire events fails at import instead of at runtime. The shape of every event stays consistent because the base defines it. And if we ever want to add a new layer of instrumentation, for example, span timing, or a structured trace ID — it's one change in the decorator, not a sweep across call sites. (I am still thinking about metrics that span across the flow; we might need to think about how to track data throughout the execution.)
The work is non-trivial because it also wants us to clean up how events are emitted today (some of the explicit calls in build.py would move into the callback methods of extracted helpers). That's why I think it belongs in its own PR.
On another note:
The dual engine path under /api/v2/workflows.
Tracing back the v2 endpoint, I noticed something worth flagging for a future conversation. Sync mode runs the graph through run_graph_internal -> graph.arun. Stream and background modes run it through _stream_event_frames -> generate_flow_events. Two different orchestrations of the same graph, behind one API.
They mostly do the same thing, but they don't share code. Vertex iteration, error construction, output collection, and event emission live in two places.
The shape I'd love us to reach is a single path through the engine where events fall out naturally as the call goes in and comes back. The graph is invoked, and a workflow_started event fires. Inside, each vertex and component announces itself as it begins and finishes. Inside those, LangChain callbacks emit their own milestones. As the call returns, events bubble back up: vertex done, component done, and finally workflow_ended (or whatever the langflow-dialect terminal event is, like end_message or something). Sync mode just doesn't ship them over the wire; stream and background do. The engine doesn't care.
That removes a whole class of "did we remember to update both paths?" bugs, and it's the natural home for the Observable pattern above. The decorator at each layer is what produces the rise-and-fall of events as the call descends and returns. Worth keeping in mind as v2 becomes the canonical execution path, the longer the two engines coexist, the more they'll drift.
I don't ask that we do this here, but should we follow up on this PR (x4) set with what I mentioned?
@ogabrielluiz @jordanrfrazier
Contributor
Author
There was a problem hiding this comment.
This is a great writeup, @dkaushik94, and I'm on board with the direction. Two things stand out to me.
The dual-engine path under /api/v2/workflows is the one I'd prioritize. Sync going through run_graph_internal -> graph.arun while stream/background go through _stream_event_frames -> generate_flow_events is exactly the "did we update both paths?" trap, and it's the same drift that's behind the Job.result question on line 977 (two paths, two result homes). Converging them so events fall out of a single engine pass, with sync just not shipping them over the wire, is the right shape. I'd like to do that before v2 hardens into the canonical path, since it only gets more expensive the longer they coexist.
The Observable base + decorator is the natural payoff once there's one engine path, and I agree it's its own PR. The thing I'd want to be careful about is custom component authors: "fails at import instead of at runtime" is a real win, but we have to make sure the contract doesn't make a third-party component harder to write. As long as inheriting is the default and emitting is automatic, that holds.
So: not in this PR, but yes, let's follow up on the stack with the engine convergence first and the Observable pattern on top of it. @jordanrfrazier does that ordering match how you're thinking about v2?
Member
There was a problem hiding this comment.
@ogabrielluiz
I think the ordering makes sense, and yes, we don't want to make DX very stringent by adding hard rules arbitrarily. Probably a warning saying, "Your component is not instrumented; to do so, please use the observable decorator and related method definitions," or something like that.
Adopt 1.11.0's composed group-mixin Settings for lfx settings/base.py (all v2 settings already present in the new groups); port the Python 3.14 cors_origins ['*']->'*' fix into groups/security.py.
…ows-bg-default Port the 6 background-execution settings and background_backend_is_scaled into 1.11.0's RuntimeSettings group mixin; union the BACKGROUND_EXECUTION_SERVICE / TELEMETRY_WRITER_SERVICE additions in schema.py and deps.py.
github-actions
Bot
added
enhancement
ogabrielluiz
requested review from
Cristhianzl,
Copilot,
erichare and
jordanrfrazier
July 8, 2026 18:35
Contributor
There was a problem hiding this comment.
Pull request overview
Implements a durable background execution service for v2 workflows (default in-process backend) by persisting job results/errors and a seq-ordered event log, adding DB-backed stop signals, and wiring restart-safe orphan reconciliation on startup.
Changes:
- Added background execution primitives (bounded executor, runner, live bus) and integrated them into
/api/v2/workflowsbackground run, stop, and events reattach flows. - Extended job persistence with
result/error, a durablejob_eventslog (UNIQUE(job_id, seq)), andexecution_signalsfor cooperative stop; added liveness-aware orphan sweeping and per-user idempotency scoping. - Added a large test suite (including a real-services tier with Postgres + Redis) and CI/Makefile wiring to validate durability and concurrency contracts.
Reviewed changes
Copilot reviewed 63 out of 66 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/lfx/tests/unit/workflow/adapters/test_event_durability.py | Tests durable vs ephemeral adapter classification |
| src/lfx/tests/unit/services/settings/test_settings_composition.py | Adds background settings keys to composition test |
| src/lfx/tests/unit/services/settings/test_background_settings.py | Tests background settings defaults + env overrides |
| src/lfx/tests/unit/services/settings/test_background_scaled_settings.py | Tests scaled-backend selection + redis test URL |
| src/lfx/tests/unit/schema/test_workflow_run_request.py | Covers new idempotency_key field round-trip |
| src/lfx/src/lfx/workflow/converters.py | Parses idempotency_key; rebuilds response from stored outputs |
| src/lfx/src/lfx/workflow/adapters/langflow.py | Defines durable milestone set + durability check |
| src/lfx/src/lfx/workflow/adapters/agui.py | Defines durable milestone set + durability check |
| src/lfx/src/lfx/workflow/adapters/init.py | Adds is_durable to adapter interface |
| src/lfx/src/lfx/services/settings/groups/runtime.py | Adds background execution runtime settings + backend selection helper |
| src/lfx/src/lfx/schema/workflow.py | Adds idempotency_key to request schema |
| src/backend/tests/unit/services/jobs/test_sweep_liveness.py | Verifies liveness-aware orphan sweep behavior |
| src/backend/tests/unit/services/jobs/test_jobs_service.py | JobService store-layer tests (result/error/events/signals/sweep) |
| src/backend/tests/unit/services/jobs/test_jobs_liveness.py | Tests heartbeat, lease staleness, atomic attempt/claim primitives |
| src/backend/tests/unit/services/jobs/test_jobs_dedupe_scope.py | Tests per-user dedupe scoping for idempotency keys |
| src/backend/tests/unit/services/jobs/init.py | Package init for jobs service tests |
| src/backend/tests/unit/services/background_execution/test_service.py | Facade-level default-backend integration tests |
| src/backend/tests/unit/services/background_execution/test_runner.py | Runner durability + terminal-state tests |
| src/backend/tests/unit/services/background_execution/test_orphan_sweep.py | Startup sweep behavior tests for queued/in-progress jobs |
| src/backend/tests/unit/services/background_execution/test_live_bus.py | Live bus subscription + replay/reattach tests |
| src/backend/tests/unit/services/background_execution/test_executor.py | Bounded executor behavior tests |
| src/backend/tests/unit/services/background_execution/conftest.py | Real-redis fixture for scaled-backend timing tests |
| src/backend/tests/unit/services/background_execution/init.py | Package init for background_execution service tests |
| src/backend/tests/unit/background_execution/test_store_real_services.py | Real-services store tests across sqlite + postgres |
| src/backend/tests/unit/background_execution/test_real_services_make_target.py | Verifies Makefile target for real-services tests |
| src/backend/tests/unit/background_execution/test_real_services_harness.py | Validates real-services harness fixtures + marker registration |
| src/backend/tests/unit/background_execution/test_real_services_ci_wiring.py | Verifies CI job wiring for real-services tier |
| src/backend/tests/unit/background_execution/test_multi_worker_boot_real_services.py | Proves multi-worker boot sweep spares live heartbeated jobs |
| src/backend/tests/unit/background_execution/test_last_event_id_contract.py | Pins Last-Event-ID namespace correctness (live vs durable) |
| src/backend/tests/unit/background_execution/test_head_to_head_deltas.py | Head-to-head “safeguard off vs on” regression proofs |
| src/backend/tests/unit/background_execution/test_executor_real_services.py | Real-services executor resilience tests |
| src/backend/tests/unit/background_execution/test_bounded_concurrency.py | Real-services bounded concurrency proof |
| src/backend/tests/unit/background_execution/test_agui_replay_byte_identical.py | Verifies replay bytes match live bytes (agui/langflow) |
| src/backend/tests/unit/background_execution/conftest.py | Real-services DB/Redis fixtures + migrations binding |
| src/backend/tests/unit/background_execution/_side_effect_component.py | Side-effect component used for exactly-once proofs |
| src/backend/tests/unit/background_execution/init.py | Package init for background_execution real-services tests |
| src/backend/tests/unit/api/v2/test_workflow.py | Updates v2 workflow tests for durable stop + error_detail |
| src/backend/tests/unit/api/v2/test_workflow_facade.py | Endpoint contract tests for facade-backed background mode |
| src/backend/tests/unit/api/v2/test_workflow_background.py | HTTP E2E tests for background submit/status/events/stop |
| src/backend/tests/unit/api/v2/test_workflow_agui.py | Adds tweaks regression test; updates background/stop semantics tests |
| src/backend/tests/unit/alembic/test_background_execution_migrations.py | Structure tests for new background-execution migrations |
| src/backend/tests/conftest.py | Registers real_services marker |
| src/backend/base/langflow/services/schema.py | Registers BACKGROUND_EXECUTION_SERVICE in ServiceType |
| src/backend/base/langflow/services/jobs/service.py | Adds durable result/error/events/signals + liveness/dedupe primitives |
| src/backend/base/langflow/services/deps.py | Adds dependency getter for BackgroundExecutionService |
| src/backend/base/langflow/services/database/models/jobs/model.py | Adds result/error, JobEvent, ExecutionSignal, SignalType |
| src/backend/base/langflow/services/database/models/jobs/init.py | Exposes new job-related models in package exports |
| src/backend/base/langflow/services/database/models/init.py | Adds new job-related models to global model exports |
| src/backend/base/langflow/services/background_execution/runner.py | Implements runner: persist durable frames + STOP reconciliation |
| src/backend/base/langflow/services/background_execution/live_bus.py | Implements in-process live bus with durable replay + tail |
| src/backend/base/langflow/services/background_execution/factory.py | Adds service factory + backend selection helper |
| src/backend/base/langflow/services/background_execution/executor.py | Adds bounded in-process executor |
| src/backend/base/langflow/services/background_execution/init.py | Background execution package init |
| src/backend/base/langflow/main.py | Triggers background orphan sweep during app lifespan startup |
| src/backend/base/langflow/api/v2/workflow.py | Wires background mode to facade; durable stop/events/status behaviors |
| src/backend/base/langflow/api/v2/workflow_execution.py | Ensures tweaks reach streaming/background build loop |
| src/backend/base/langflow/api/build.py | Applies request tweaks in streaming/background graph builds; gates memory hook |
| src/backend/base/langflow/alembic/versions/b026885b89c8_add_job_events_table.py | Migration: add job_events table |
| src/backend/base/langflow/alembic/versions/8ce44e4858c6_add_execution_signals_table.py | Migration: add execution_signals + enum |
| src/backend/base/langflow/alembic/versions/185482a2d715_add_result_error_to_job.py | Migration: add job.result and job.error |
| pyproject.toml | Registers real_services marker |
| Makefile | Adds real_services_tests target |
| .secrets.baseline | Updates baseline after workflow allowlist pragmas |
| .github/workflows/migration-validation.yml | Adds background real-services CI job (Postgres + Redis) |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+165
to
+170
| if item is _CLOSED: | ||
| return | ||
| if item.seq <= highest: | ||
| continue | ||
| highest = item.seq | ||
| yield item |
Comment on lines
+804
to
808
| # in-flight runner racing to a terminal state reliably observes the stop | ||
| # and finalizes CANCELLED rather than overwriting it with COMPLETED/FAILED. | ||
| with contextlib.suppress(Exception): | ||
| await get_background_execution_service().stop_job(job_id, current_user) | ||
| await job_service.update_job_status(job_id, JobStatus.CANCELLED) |
ogabrielluiz
added a commit
that referenced
this pull request
Jul 8, 2026
The base got its own release-1.11.0 merge (#13507), resolved independently of this branch's, so the two collided on the files both touched. Resolutions: workflow.py keeps this branch's HITL routes and job_id/resume threading; the build.py and workflow_execution.py conflicts were comment wording only, taking the base's. .secrets.baseline regenerated.
ogabrielluiz
enabled auto-merge
July 8, 2026 18:55
github-actions
Bot
added
enhancement
…copg in real-service tests The real-service CI job broke on `ModuleNotFoundError: No module named 'asyncpg'`. asyncpg was never declared anywhere in this repo; it arrived transitively through `cuga`, which release-1.11.0 dropped when it moved the root dep to `lfx-bundles[all-no-torch]` (#13886). Normalize the harness URL to `postgresql+psycopg` instead: it is what `--extra postgresql` installs and what DatabaseService already selects for async Postgres, so the test now exercises the production driver rather than a stowaway. Also fix a real reattach bug. The runner publishes ephemeral token frames tagged with the last durable seq (they have no job_events row), while reattach's tail skipped anything with `seq <= highest`. After replay left `highest` at that same seq, every token delta was dropped until the next durable milestone advanced it, so a reconnect mid-stream saw no tokens. Mark frames durable/ephemeral and dedupe only the durable ones, which are the only frames a replay can return. Log instead of silently suppressing a failed stop_job signal, and drop a comment block duplicated verbatim in InProcessExecutor.stop().
github-actions
Bot
added
enhancement
github-actions
Bot
added
enhancement
dkaushik94
approved these changes
Jul 10, 2026
github-actions
Bot
added
enhancement
jordanrfrazier
pushed a commit
that referenced
this pull request
Jul 13, 2026
…end) (#13507) * feat: native v2 workflows endpoint with pluggable stream protocols Rebased onto release-1.10.0. The base independently rebuilt the v2 workflows backend (RBAC, body globals, share-aware fetch); keep our forward design and conform its auth to that work: 1. Auth: keep get_current_user_for_workflow (session-or-API-key authN that does not hold a DB connection during the inline run, avoiding the SQLite lock contention api_key_security would cause) and enforce the base's RBAC on top: ensure_flow_permission(EXECUTE) before run, (READ) before status reconstruct, with widen_for_shares fetch. 2. Port the base's request-body globals onto the v2 WorkflowRunRequest. The X-LANGFLOW-GLOBAL-VAR-* headers stay supported (the Responses API passes globals that way); body globals win on conflict. Converters echo the effective globals via effective_globals. 3. Public endpoint keeps the v1 build_public_tmp posture (access_type==PUBLIC, run-as-owner); RBAC applies to the authenticated endpoint only. 4. Preserve the base's post-build KB-cache invalidation in the AG-UI build path. The endpoint, AG-UI bridge, pluggable stream adapters, public endpoint, and re-attach are unchanged. * feat(api): add output_text and session_id to v2 workflow response The synchronous /api/v2/workflows response keyed every result under its component id, so reading the answer meant knowing an id you can't predict. Surface two additive fields: - output_text: the flow's single text answer (ChatOutput/TextOutput). None when the flow has zero or multiple text outputs, so callers read outputs rather than the shortcut guessing which channel is the answer. - session_id: echoes the resolved session so chat/memory callers can continue the same thread (v1 /run returned this; v2 had dropped it). outputs is unchanged, so this is non-breaking. * test(api/v2): cover output_text and session_id on the v2 workflow response Pin the sync-response shortcuts on the v2 workflows endpoint: - output_text surfaces the lone ChatOutput/TextOutput text and stays None for non-output message nodes, data-only flows, and multi-text flows - session_id echoes the resolved session; the error response exposes neither - each outputs entry exposes only {type, status, content, metadata}, with the component id carried by the dict key Also drop the component_id kwarg the converter passed to ComponentOutput, which has no such field and silently dropped it. * feat(api/v2): structured output with resolution reason on v2 response Replace the flat output_text shortcut with an `output` object carrying the resolved text answer plus a `reason` that explains why it resolved that way (single/multiple/none/non_string/failed), so a null answer is always diagnosable instead of silently None. `reason` follows the LLM-domain finish_reason/stop_reason convention, distinct from the lifecycle status. Also add `display_name` to each ComponentOutput (the stable component id stays the dict key) and a computed `has_errors` flag derived from errors. * feat(api/v2): add request-side output selection (output_ids) Let a sync caller name the output(s) they want via output_ids so output.text resolves deterministically (reason=single) on multi-output flows instead of going null. Selection is steer-only: it picks the answer among the named outputs without filtering the outputs map. Invalid ids are rejected with 422 before the flow runs (and before any job row is created), so a typo costs no compute. Resolution considers selected outputs that actually fired, so branching flows resolve to whichever candidate ran. * feat(api/v2): emit per-output events on the langflow stream Give v2-workflows sync and the langflow stream protocol one parser. The stream now emits a normalized "output" event per terminal output carrying an OutputEvent (the ComponentOutput shape sync returns in outputs[id], plus component_id). A shared build_component_output() backs both the sync converter and the adapter, and the build loop ships authoritative vertex metadata as an additive output_meta key on end_vertex (existing consumers read build_data and ignore it). This is access-pattern parity (one parser, same fields, same terminal set), not byte-identical content: the stream reuses the v1 build path whose display serialization differs from sync's run_graph output. * feat(api/v2): durable background execution service (store + default backend) Turns v2 mode:background into a durable, in-API background execution service behind a BackgroundExecutionService facade. Adds the store layer (result/error columns, job_events durable milestone log, execution_signals control, heartbeat/lease, 3 migrations), the default backend (bounded executor, runner, in-memory live bus, liveness-aware single-flight orphan sweep), the v2 endpoint rewiring, and the real-instance test harness. Needs no new infra; works on the SQLite single-process install. The redis-scaled worker backend is stacked on top in a follow-up PR. * test(background-execution): rename hard_proof marker to real_services The hard_proof marker name was a vibe word that said nothing about what the tests need. Rename it to real_services everywhere: the pytest marker registration, the *_hard_proof.py test files, the Makefile target (real_services_tests), the -m selector in migration-validation.yml, and the CI job. real_services says what these tests require: real Postgres + Redis + worker subprocesses. (integration was already taken for the external-API suite under tests/integration.) * fix(api/v2): enforce no-code-execution gate on public workflow endpoint The v2 public endpoint only ran validate_flow_for_current_settings and skipped validate_public_flow_no_code_execution, which the v1 build_public_tmp path applies. A public flow containing a Python interpreter/REPL (or the legacy Python Code Structured tool, Smart Transform lambda) was therefore an unauthenticated server-side code-execution primitive (report H1-3754930). Mirror v1: import the validator and call it right after the public-access gate. PublicFlowValidationError subclasses CustomComponentValidationError, so the existing handler already sanitizes it to a 400 'This flow cannot be executed.' without leaking the blocked component class names. Add a non-mocking test that builds a public flow with a real PythonREPLComponent and asserts the sanitized 400 (verified RED: returns 200 without the gate). LE-1389 * fix(api/v2): reconstruct background workflow status from job-keyed vertex builds A completed background job's GET status 500'd with 'No vertex builds found for job_id'. The background build path differed from the sync path twice: 1. generate_flow_events minted a fresh run_id instead of using job_id, so vertex builds were keyed by an id the status query never uses. Thread run_id through _stream_event_frames -> generate_flow_events and pass job_id from the background buffer so graph.run_id == job_id (the sync path already does graph.set_run_id(job_id)). 2. The SSE build loop (build_vertices) only persisted builds when log_builds was set and never passed job_id. Tie log_builds to job-tracked runs (run_id present) and pass job_id=graph.run_id on the persist call. Job-tracked runs also persist streaming terminal vertices so reconstruction is complete; the live build path (run_id is None) keeps its original behavior, so the v1 build path is unchanged. Test: a real background run polled to completion, then GET status asserts a reconstructed 200 (verified RED: 500 'No vertex builds found' before the fix). Covers the non-streaming flow. v1 build path unchanged (35 build tests pass); AG-UI suite 46 pass. LE-1389 * fix(api/v2): merge workflow AG-UI cancellation hardening LE-1389 * fix(api/v2): signal cross-worker workflow stops LE-1389 * fix(api/v2): report unconfirmed workflow stops LE-1389 * fix(api/v2): keep background workflows out of polling watchdog LE-1389 * fix(api/v2): buffer parallel messages in the AG-UI translator instead of dropping them Parallel components stream tokens for different message ids interleaved. The translator tracked a single open message: the first foreign token closed the open message and tombstoned its id, so every later event for it was dropped and its remaining text never reached the client. Tokens for a message that cannot take the wire now buffer until the open message genuinely ends (its add_message finalizer), then flush in arrival order; complete messages landing mid-stream buffer the same way instead of interleaving a second START. end/error drain all buffers before the terminal event. The wire still carries at most one open text message, so the stream stays AG-UI-conformant. * fix(api/v2): gate AG-UI message finalization on non-partial state and purge removed buffers A partial add_message re-fire (the agent path emits these at tool start/end for a message it is still streaming) was treated as the finalizer: it closed and tombstoned the id, so the post-tool answer was dropped. Only a non-partial add_message finalizes now; state defaults to complete, so payloads without properties are unchanged. remove_message now purges a buffered message and tombstones its id, so text the backend retracted is not flushed to the client later. * Fix AG-UI workflow lifecycle edges * [autofix.ci] apply automated fixes * fix(frontend): enable downlevelIteration for jest Set/Map iteration ts-jest compiles with target es5; without downlevelIteration, [...set] and for...of over a Set/Map emit ES5 that yields nothing. That silently broke the AG-UI bridge tests: runningNodeIds spread, markRunningNodesFailed, and restoreOriginalBuildStatuses all iterated empty. Production (Vite/SWC, modern target) was never affected; only the ts-jest harness was. Fixes the 3 failing jest tests on this branch with no other suite changes (4994/4994 pass). * fix(api/v2): surface inactivated branch vertices over AG-UI A branch component (If-Else, Conditional Router) reports its not-taken vertices in build_data.inactivated_vertices, but the AG-UI translator only emitted the branch node's own success/error status and dropped that list. The canvas seeds every planned node as pending from vertices_sorted; skipped vertices then get no build_start/end_vertex, so they stayed stuck on pending instead of rendering as inactive (the v1 build path marked them INACTIVE). The translator now appends an inactive STATE_DELTA op per inactivated vertex, and the frontend bridge maps the new inactive status to BuildStatus.INACTIVE and tears its edges down like a completed node. Fixes the If-Else regression in general-bugs-reset-flow-run.spec.ts. * fix(api/v2): dedupe repeated inactive node deltas in AG-UI stream build.py keeps reporting a conditionally-excluded vertex in inactivated_vertices on every subsequent end_vertex (the excluded set persists until the ConditionalRouter clears it), so the translator was putting the same inactive STATE_DELTA on the wire once per remaining vertex. Track emitted inactive nodes and skip re-emitting; drop a node from the set when it actually runs again (build_start/end_vertex) so a loop re-activation can still re-emit inactive later. * fix(api/v2): no duplicate WORKFLOW job row on durable background runs A v2 background run created TWO JobType.WORKFLOW rows for one flow execution: the durable row (submit()'s job_id, owned by JobRunner) plus an orphan keyed by the run_id generate_flow_events mints, because the build pipeline's track_job_status defaults True and the durable frame source never passed False. The flow ran once (double bookkeeping), but every background run left a phantom WORKFLOW row + job_events and double-fired the memory-base hook, skewing metrics. Thread track_job_status through _stream_event_frames; pass False only from the background frame source (the durable runner already owns the row + fires the hook with the durable job_id). Stream/public paths keep default True. Also gate build.py's memory-base hook fire behind track_job_status so background doesn't double-fire. Adds a regression test (RED before fix: found 2 rows). * test(api/v2): update stale workflow-stop tests for the durable design These 3 tests targeted the removed queue-service stop helper (_cancel_workflow_queue_job / get_queue_service), inherited via the agui->bg-default merge and failing with AttributeError across the stack: - test_stop_workflow_success: adapted to the durable stop path (revoke_task -> stop_job -> update_job_status(CANCELLED)). - test_stop_workflow_allowed_for_legacy_job_with_no_user_id (IDOR): adapted to the durable mechanism; still asserts the ownership check does not block a legacy user_id=None row. - test_stop_workflow_returns_503_when_queue_cancel_cannot_be_confirmed: dropped — the durable stop writes a best-effort STOP signal and always finalizes CANCELLED; the queue-service 'cannot confirm -> 503' path no longer exists. test_workflow.py now passes 24/24. * fix(api/v2): move FrameSourceFactory alias under TYPE_CHECKING As a module-level runtime value, FrameSourceFactory = Callable[..., Any] is a GenericAlias that passes isinstance(obj, type) but makes issubclass(obj, Service) raise on Python 3.10/3.14. The service factory scans this module for Service subclasses (services/factory.py:90), so the runtime alias crashed service initialization on those interpreters with 'issubclass() arg 1 must be a class' -> 'Could not initialize services', erroring out dozens of unrelated tests at setup (3.13 was unaffected, which is why local runs passed). The alias is only referenced in a lazy annotation (from __future__ import annotations), so moving it + the Callable import under TYPE_CHECKING removes it from the runtime namespace with no behavior change. Verified: alias absent from runtime module namespace, factory scan finds BackgroundExecutionService cleanly, durable service tests 26/26 pass. * test(lfx): register background-execution Settings fields in the field-count gate The durable background-execution work added six Settings fields (background_max_concurrency, background_job_timeout, background_lease_ttl_s, background_heartbeat_interval_s, background_watchdog_interval_s, test_redis_url) without updating EXPECTED_FIELDS, so test_field_count_unchanged failed 152 != 146. These are intentional bg-exec config; add them to the gate. * fix(api/v2): restore "end" side-channel event in AG-UI workflow stream The durable background-execution rewrite of workflow.py reverted `side_channel_events` to its pre-"end" form, dropping the "end" event from the AG-UI side-channel. That event carries `build_duration` to the playground chat-view, and the message metadata badge only renders when `hasDuration || hasTokens`. With build_duration gone the badge vanished, failing the token-usage and shareable-playground "Finished In" regression tests. Re-add "end" so the streaming playground path delivers it again. * fix(api/v2): apply request tweaks on the streaming and background paths The v2 workflows endpoint applied `tweaks` only on mode=sync. The stream and background paths build the graph via the v1 build-vertex loop (`generate_flow_events`), which never received the tweaks, so they were silently dropped. The confusing symptom: a model passed via tweaks surfaced as "A model selection is required", and any per-component override was ignored on non-sync runs. Thread `parsed.tweaks` into `generate_flow_events` and apply them to the built graph via `vertex.update_raw_params`. We do not use the lfx `process_tweaks_on_graph` helper because it only sets `vertex.params`, which does not persist to runtime (the same bug `lfx.base.tools.run_flow._process_tweaks_on_graph` works around). No-tweaks runs are unchanged (guarded by `if tweaks`). Adds a streaming regression test that overrides ChatInput via tweaks and asserts the value drives the run. * fix(api/v2): return background run output from completed status A completed background run's GET status returned a bare COMPLETED with an empty `outputs` and a null `output`. The COMPLETED branch reconstructs from `vertex_builds` keyed by job_id, which the durable path does not write, so reconstruction raised ValueError and fell through to an empty response; the result was only retrievable via a /events re-attach. The runner now captures the terminal `output` events (the langflow adapter's normalized ComponentOutput payloads) into `Job.result`, and the status COMPLETED branch rebuilds the `outputs` map and resolved `output` from them via `workflow_response_from_output_events`, matching the sync response. agui-protocol runs emit no `output` events, so their status stays result-less (the result remains on the /events log). * fix(api/v2): address review findings on the v2 workflows endpoint - recover session_id for completed background jobs from the persisted terminal message instead of always returning null, so GET status can continue the same chat/memory thread - replay a user-cancel as a CUSTOM cancel marker + RUN_FINISHED (agui) and a `cancelled` terminal (langflow) instead of RUN_ERROR, so a re-attaching client no longer reads a deliberate stop as a failure - cancel the evicted still-running buffer writer when the background-run registry is full, so it stops appending into a run no reader can find - derive per-component status from the error artifact / valid flag instead of hardcoding COMPLETED, and stop the langflow adapter dropping `valid` - throttle the unauthenticated public endpoint per IP and bound its input_value/session_id length - document the sync-only scope of request-body globals - document that live event re-attach is intentionally owner-only * test(lfx): register public_flow_rate_limit_per_minute in settings composition * refactor(v2 workflows): split workflow.py and address review blockers Splits the ~1.5k-line workflow.py into focused modules and folds in the execution-timeout and error-sanitization fixes from Cristhianzl's review of #13307. - B1: workflow.py now holds only the four route handlers. Validation guards move to workflow_validation, the sync/stream run loop to workflow_execution, and the durable background machinery to workflow_background (layered, acyclic). - I1: add workflow_execution_timeout (default 300) and apply a single wall-clock ceiling across sync, stream, background, and public via _stream_event_frames. A timeout becomes a sanitized terminal error and marks a background job failed. - I3: the route error handlers no longer echo raw exception text. They return a generic, code-tagged message and log the full exception server-side. - R1: remove the "commented out / future scope" comments that sat over live dataframe-extraction code in converters.py. - R4: drop the worker-routing internals from the reattach 409 message. Tests cover the timeout terminal-error path and the error-body sanitization, and the settings field-count guard is updated for the new setting. * refactor(lfx): extract v2 workflow contract layer into lfx.workflow Moves the protocol-agnostic pieces of the v2 workflows API out of the langflow backend into lfx so both the backend and `lfx serve` can share one contract. First step toward giving lfx (the production runtime) the v2 workflows API. - Move api/v2/adapters/, agui_translator.py, and converters.py to lfx/workflow/. They depend only on lfx.schema.workflow and ag_ui (already an lfx dep), so lfx carries the contract with zero langflow imports. - Decouple the one langflow reference: converters typed run_response against langflow.api.v1.schemas.RunResponse (TYPE_CHECKING only). Replaced with a local RunResponseLike Protocol (outputs + session_id), the only attributes used. - Repoint the six backend v2 workflow modules to import from lfx.workflow. - Move the five protocol-agnostic contract tests into src/lfx/tests/unit/workflow/ (run in the lfx-only env). test_output_event_parity and test_workflow_agui stay in langflow (they need langflow.api.build) with repointed imports. Coverage unchanged: 201 contract tests pass in the lfx-only env, 191 backend v2 tests pass; 392 total, same as before the move. * fix(background-execution): prevent worker deadlock on stop() under Python 3.10 The bounded executor's worker awaited the in-flight job task with a bare `await task`. On Python 3.10, when stop() cancels a worker while its job task is finishing, the awaiter's wakeup is lost and the event loop idles forever in select(), deadlocking stop(). Await via a done-callback Event (the same mechanism stop()'s own asyncio.gather already uses), which delivers the wakeup reliably; task.result() preserves the cancellation and exception semantics of the bare await. * fix(background-execution): keep ephemeral frames on reattach, pin psycopg in real-service tests The real-service CI job broke on `ModuleNotFoundError: No module named 'asyncpg'`. asyncpg was never declared anywhere in this repo; it arrived transitively through `cuga`, which release-1.11.0 dropped when it moved the root dep to `lfx-bundles[all-no-torch]` (#13886). Normalize the harness URL to `postgresql+psycopg` instead: it is what `--extra postgresql` installs and what DatabaseService already selects for async Postgres, so the test now exercises the production driver rather than a stowaway. Also fix a real reattach bug. The runner publishes ephemeral token frames tagged with the last durable seq (they have no job_events row), while reattach's tail skipped anything with `seq <= highest`. After replay left `highest` at that same seq, every token delta was dropped until the next durable milestone advanced it, so a reconnect mid-stream saw no tokens. Mark frames durable/ephemeral and dedupe only the durable ones, which are the only frames a replay can return. Log instead of silently suppressing a failed stop_job signal, and drop a comment block duplicated verbatim in InProcessExecutor.stop(). * fix(migrations): merge alembic heads (mcp_server + execution_signals) --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
erichare
added a commit
that referenced
this pull request
Jul 13, 2026
…rve (#13818) * feat(lfx): per-request-isolated flow execution for lfx serve Run `lfx serve --workers N` flows under per-request process isolation so cross-request os.environ credential leakage is structurally impossible, while sharing the warm library + flow graphs across workers via COW. - gunicorn --preload + UvicornWorker + max_requests=1: the master builds and warms the FlowRegistry once and forks workers (copy-on-write); each worker is recycled after one request. - serve_preloaded_app.py: import-time master entrypoint (build_registry_from_env + gc.freeze) inherited by forks. - serve_gunicorn.py: LFXGunicornApp process manager. - _EXECUTE_GUARD asyncio.Semaphore(1) + guarded_execute around execute_graph_with_capture (run + stream), closing the async concurrent-overlap window that max_requests=1 alone leaves open. - commands.py: _launch_workers launches gunicorn on Unix; refuses --workers>1 on Windows (gunicorn is Unix-only). - credentials.py: gate the post-DB-miss SECRET_KEY-rotation env fallback behind is_env_fallback_disabled() so no_env_fallback is honored (A3). - FlowRegistry.get logs store reconstruction so per-request rebuild cost of uploaded flows is observable. - gunicorn>=22.0 added as a platform-gated (non-Windows) lfx dependency. Tests: process-model (preload/warm/guard/fork-safety), synchronous run contract, no_env_fallback credential regression; existing multi-worker serve tests migrated to the gunicorn launcher. 173 serve+credentials tests green. * Add pre-fork hook * feat(lfx): add opt-in --reset-environ and --sync-workers to lfx serve Two opt-in flags for per-request isolation in multi-worker lfx serve, both defaulting to the existing committed behavior: - --reset-environ: snapshot/restore os.environ around each flow run so a flow's env mutations (or request-scoped credentials) cannot leak into the next request served by the same warm worker (gated by LFX_SERVE_RESET_ENVIRON; read per request in guarded_execute). - --sync-workers: serve via gunicorn's blocking sync worker behind an a2wsgi ASGI->WSGI bridge so the kernel routes each request to an idle worker (one whole request per worker at a time) instead of queueing behind an in-flight request on a busy async worker. The bridge is built lazily post-fork; refused on Windows. Also fix LFXGunicornApp.load() to honor the app import string (it hardcoded the ASGI app, which fed the WSGI sync worker the wrong callable), and declare a2wsgi as a Unix-only dependency alongside gunicorn. * feat(lfx): add --timeout flag for lfx serve worker timeout Expose gunicorn's worker timeout (previously hardcoded at 120s) as --timeout, on both serve_command and the CLI wrapper. A worker that doesn't complete a request within --timeout seconds is killed and restarted. Matters most under --sync-workers: a blocking sync worker cannot heartbeat mid-request, so long-running flows need a higher timeout. Default stays 120s (unchanged behavior); no effect on the Windows uvicorn fallback. * fix(lfx): register VariableService in serve so request-scoped global_vars resolve lfx serve never registered a VariableService, so get_variable_service() returned None in the worker. Every credential path that resolves through it (get_api_key_for_provider, get_all_variables_for_provider, model_utils, KB connectors) therefore never consulted the request scope and fell back to os.environ -- which --no-env-fallback then blocks. As a result, request-scoped global_vars could not supply a model/agent provider API key unless the flow's api_key field was explicitly wired to load the variable. Register the minimal in-memory VariableService in create_multi_serve_app (idempotent; a real DB-backed service is left untouched), making all of those paths request-scope-aware at once. Guard the MCP component's DB-only get_all_decrypted_variables call with hasattr so the minimal service is skipped cleanly instead of raising into its broad except. Verified end-to-end: Basic Prompting and Simple Agent starter flows execute successfully under --no-env-fallback with the OpenAI key supplied only via global_vars, while a request without global_vars still fails (no env leakage). * refactor(lfx): rework lfx serve worker/concurrency flags - Remove --limit-concurrency; --use-sync-workers covers bounded concurrency - Rename --sync-workers to --use-sync-workers/--use-async-workers (default async) - Default --max-requests to periodic recycling (~1000, 10% jitter) for memory hygiene; 0 disables, explicit N overrides - Make Windows/a2wsgi flag refusals loud (typer.echo, not verbose_print) - Declare uvicorn[standard] so standalone lfx installs get uvloop/httptools - DRY: shared serve --help strings (_serve_help) + env context manager in _launch_workers - Document --use-sync-workers / --reset-environ as the per-request isolation mechanisms (async --max-requests is worker hygiene, not isolation) * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * fix(lfx): warn on --timeout with single worker; drop dead limit-concurrency env - Make --timeout an int|None sentinel (matching --max-requests) so the single-worker path can warn that it is ignored instead of silently honoring it; None resolves to DEFAULT_TIMEOUT (120) for gunicorn. - Remove the unused _SERVE_LIMIT_CONCURRENCY_ENV constant. - Dedup the FakeGunicornApp test stubs behind _make_fake_gunicorn_app. * update dep docs * simplify cmmments * skip gunicorn threads in ghost finder * chore: auto-bake note keys and regenerate backend locales/en.json [skip ci] * fix(lfx): address review comments on serve worker flags - normalize reset_environ/sync_workers OptionInfo sentinels for direct serve_command() callers (truthy sentinels silently enabled the flags) - _exported_env now restores overwritten env vars instead of deleting them - clarify MAX_REQUESTS help: strict isolation needs --reset-environ, or --use-sync-workers WITH --max-requests 1 (sync worker alone is not isolation) - correct Windows fallback comment in pyproject - assert LFX_SERVE_RESET_ENVIRON cleanup; reset preloaded module state in test - add synchronous error-contract test for /run * chore: auto-bake note keys and regenerate backend locales/en.json [skip ci] * fix(lfx): close reset-environ auth race and warn on unisolated multi-worker - guarded_execute restores os.environ by diff instead of clear()+update(): clear() empties key-by-key and could race a threadpool-run verify_api_key into a 401 - snapshot the API key at app startup (app.state) so auth never reads live os.environ per request; lazy-cache fallback preserves post-creation config (tests) - warn when --workers > 1 has no per-request isolation (no --reset-environ and not --use-sync-workers with --max-requests 1) - wrap the gunicorn import with a friendly error, matching the a2wsgi check - xfail(strict=False) for the timing-dependent async-recycle leak test - --workers help: gunicorn on Unix, uvicorn on Windows - tests: diff-restore add/change/delete + no-clear lock, cached-key auth, isolation warning * chore: auto-bake note keys and regenerate backend locales/en.json [skip ci] * fix: prevent LangfuseResourceManager deepcopy failure in Agent/Tool Calling Agent (#13966) * fix: prevent LangfuseResourceManager deepcopy failure in Agent components Adds __deepcopy__ to _RootRunReparentingCallbackHandler so that deepcopy(component) calls in component_tool.py no longer trigger LangfuseResourceManager.__new__() with no credential kwargs. Root cause: the #13429 fix (c4d27818a) introduced _RootRunReparentingCallbackHandler as a CallbackHandler subclass. CallbackHandler internally holds a LangfuseResourceManager whose __new__ now requires explicit public_key/secret_key/base_url in newer langfuse builds. Python's deepcopy calls __new__ with no args during copy construction, raising: LangfuseResourceManager.new() missing required keyword arguments The handler carries no per-invocation mutable state, so returning self from __deepcopy__ is safe. The same root cause and workaround are already documented in flow_loader.py (agentic path) for issue #13429. Adds a focused regression test asserting deepcopy(handler) is handler and does not raise. Fixes #13965 * fix(ruff): replace unused lambda args a/kw with _ to fix ARG005 * fix(ci): publish bundle deps before nightly main (#13982) * fix(ci): publish bundle deps before nightly main * fix(ci): wait for bundle pypi propagation --------- Co-authored-by: Eric Hare <ericrhare@gmail.com> * feat(mcp): persist MCP servers in a database table (#13976) * feat(mcp): persist MCP servers in a database table MCP servers were stored in a per-user JSON file guarded by an in-process lock, so concurrent edits lose updates and the list diverges across workers/replicas. Store them in a new mcp_server table (one row per server, unique on (user_id, name)); get_server_list/get_server/update_server become per-row reads/upserts and the in-process lock is removed, so MCP config is safe at any worker/replica count. Secret values (env/headers) are encrypted at rest. Existing _mcp_servers_*.json files are imported by an idempotent startup backfill and a `langflow migrate-mcp` CLI; the file is kept for rollback. REST shape and function signatures are unchanged. Fixes #13970 * fix(mcp): clear hashed cache-key variants when a server changes The shared tool cache is keyed {server_name}:{hash of headers+timeout} (MCPComponent._mcp_servers_cache_key), so clearing only the bare server name left servers with custom headers/timeouts serving stale config after an update. Clear the bare name and all {server_name}: variants. * fix(mcp): re-apply create/merge rules after IntegrityError refetch The IntegrityError fallback in update_server reused the pre-race config, so a concurrent same-name create (check_existing) could overwrite the winning row and a concurrent PATCH (merge_existing) skipped the merge. Now re-apply the rules against the refetched row: check_existing raises 'already exists', merge_existing merges, otherwise overwrite. Adds a concurrency test. * fix(mcp): order get_server_list by created_at to preserve insertion order get_server_list had no ORDER BY, so with several rows the server list order was undefined. The legacy file (a dict) preserved insertion order and the UI relies on it (the starter project must stay first). Order by created_at to restore stable insertion order. * fix(mcp): re-point migration onto current base head to resolve alembic multi-head release-1.11.0 gained migration c3e7a1b9d2f4 after this branch opened. Our migration also chained off 4f0d2c9a8b7e, producing two alembic heads so 'alembic upgrade head' failed and the DB never initialized (cascading to all e2e/integration/docker jobs). Re-point down_revision from 4f0d2c9a8b7e to c3e7a1b9d2f4 so the chain is linear with a single head. --------- Co-authored-by: Saad ur Rehman <saad.urrehman@cleura.com> * docs: external authentication, RBAC, SSO (#13866) * docs: add rate limiting and signup env vars * docs: configure global vars in k8s secrets * docs: per-process login rate limiting * docs: initial rbac and sso content * docs: move jwt page to auth and add redirect * docs: add keycloak example * docs: authorization plugin interface * fix(docker): give runtime user a writable npm cache so stdio MCP servers start (#13992) * fix(docker): give runtime user a writable npm cache so stdio MCP servers start The ubi10 runtime base sets HOME=/opt/app-root/src, and the main image runs `npm install -g` as root, which seeds a root-owned npm cache at /opt/app-root/src/.npm. The runtime user (uid 1000) spawns stdio MCP servers via `npx`, which then fails with `EACCES` on ~/.npm/_cacache, so every stdio MCP server registers but never lists any tools (toolsCount stays null in GET /api/v2/mcp/servers?action_count=true). Pin npm's cache to a uid-1000-owned directory via NPM_CONFIG_CACHE=/app/.npm (immune to the base image's HOME) and hand ownership of the default HOME cache to the runtime user as a fallback. Applied to all three runtime images (main, with_extras, base) that ship Node so npx has a writable cache. * fix(docker): avoid incompatible npm latest * fix: thin single focus border on global-variable input (#13995) fix input style on GV input * ci: add non-blocking flaky test report job with 30-day retention (#13996) * ci: add non-blocking flaky test report job with 30-day retention * fix: tolerate partial blob downloads in flaky-report job One of the ~62 shard blob artifacts failed its download after 5 retries (transient artifact storage error), which failed the whole download step and skipped the merge. For a metrics-only job, merging the shards that did download beats producing nothing: mark the download step continue-on-error and no-op the merge cleanly when no blobs landed. * fix(a11y): improve API keys tab order (#13953) * fix(frontend): repair flow list card a11y * fix(frontend): clear flows page a11y debt * test(frontend): update header count a11y names * test(e2e): open flow cards via action button * fix(a11y): address flow card review * chore: update secrets baseline * fix(a11y): improve settings table scans * refactor(a11y): isolate table scan fixes * fix(a11y): improve API keys tab order * test(frontend): fix shard click targets * fix(a11y): link API key expiry label * fix(a11y): resolve AG Grid accessibility bugs in table component - Fix hidden row control restoration bug by tracking original tabindex values - Fix pagination button self-latching bug by using only class as input signal - Add comprehensive unit tests for both fixes (10 tests, all passing) - Export patchGridAccessibility function and AgGridAccessibilityLabels type for testing - Improve api-keys.a11y.spec.ts to be less brittle with dynamic checks Addresses feedback from keval718 on PR #13953 * fix(a11y): rework API keys table keyboard nav and clear IBM violations Replace the AG Grid DOM-patching hack (MutationObserver + global keydown/ focusin listeners + role rewriting + multi-timeout retries) with a minimal, event-driven patch applied via the grid's own events. Keyboard/tab order: - Tab from the last cell reaches the next control in one press (was three via two dead <body> stops) using AG Grid's tabToNextCell hook. - Disabled pagination buttons are no longer tab stops, and a container-scoped focusin redirect keeps AG Grid from programmatically focusing them; inert/ disabled attributes are avoided because they break the tab guards and trap reverse (Shift+Tab) entry into the grid (WCAG 2.1.2). - API key name cell opens by keyboard (Enter/Space). IBM Equal Access: 8 -> 0 violations on /settings/api-keys (and the shared /settings/global-variables table): - tab guards given role=button + label (element_tabbable_role_valid, 4.1.2) - empty rowgroups demoted to role=presentation (aria_child_valid, 1.3.1) - first body row made tabbable via roving tabindex (aria_child_tabbable) Also: ProviderListItem and the text-cell modal trigger become real buttons, and Shortcuts settings cells open by keyboard. Tests: Playwright single-Tab exit + reverse-tab re-entry regression guards, tab-order test updated to the roving-tabindex model, and Jest unit tests for the tab-guard / rowgroup / roving-row / disabled-paging patches. * fix(frontend): center update toast buttons vertically (#14004) * docs: lfx user docs and bundle extension changes (#13775) * docs: add LFX user documentation section Creates a new Lfx/ folder in the docs with three pages covering the LFX executor from a user perspective: - lfx-overview.mdx: what LFX is, when to use it, and a full command reference table linking to the DevOps SDK docs - lfx-install.mdx: install from PyPI, with bundles, from source, or via uvx — plus bundle install guidance for standalone lfx users - lfx-run.mdx: lfx serve and lfx run with full option tables, stdin/inline JSON, Python script usage, and component category allowlist/blocklist controls Also adds an LFX sidebar category in the "Develop & Deploy" section and a tip in flow-devops-sdk.mdx noting that standalone lfx installs do not include bundle package components. Co-authored-by: Cursor <cursoragent@cursor.com> * docs: move Flow DevOps SDK from API Reference to LFX sidebar * docs: add terminal sidebar icon for LFX section * docs: move Flow DevOps SDK page into Lfx folder * docs: move extensions to lfx section * docs: clean up lfx overview * docs: extension manifest update and release notes * docs: revert release notes * docs: add back nextplaid bundle to release notes * docs: move mcp and compatibility, add prewarm * docs: fix relative link * docs: partials for lfx-bundles * docs: clarify bundle differences * docs: remove lfx prewarm * docs: fix broken lfx user docs link * docs: lfx bundle installation * docs: update graduated bundles * docs: torch opt-in changes * docs: fix partial link * docs: bundles some small fixes * docs: allowlist and blocklist for lfx * docs: separate run and serve pages * docs: final extensions check * docs: lfx MCP page instructions --------- Co-authored-by: Cursor <cursoragent@cursor.com> * fix(a11y): make /assets/files accessible (#13987) * fix(a11y): make /assets/files accessible (LE-1744) Bring the Files page to WCAG 2.1 A/AA, verified with IBM Equal Access plus keyboard/focus tests. - Name the icon-only actions column header, hidden visually (4.1.2) - Open the row actions menu by keyboard from the grid cell (2.1.1) - Label the upload/bulk-delete buttons so they are named on mobile (4.1.2) - Restore a keyboard-only :focus-visible ring on borderless grid cells (2.4.7) - Merge the bulk-delete dialog trigger via asChild to remove the duplicate tab stop (2.4.3) - Move focus into the cell editor when renaming (2.4.3) - Make the upload "try again" retry a real button (2.1.1) Add a dedicated Playwright a11y suite (files.a11y.spec.ts) covering the full state matrix and keyboard/focus behaviour, move the IBM baseline folder under tests/a11y, and baseline the known Radix menu-portal landmark item. * feat(a11y): add frontend accessibility check skill documentation * fix(a11y): return focus to upload button after file picker cancel (LE-1744) Keyboard-activating Upload Files then dismissing the native picker with Esc dropped focus to <body> (WCAG 2.4.3). Only blur on mouse clicks (to keep the #13178 tooltip fix); restore focus to the trigger on keyboard activation. Adds two behavioural a11y tests. * feat(playground): content blocks frontend renderer for v2 workflows (#13391) * feat: native v2 workflows endpoint with pluggable stream protocols Rebased onto release-1.10.0. The base independently rebuilt the v2 workflows backend (RBAC, body globals, share-aware fetch); keep our forward design and conform its auth to that work: 1. Auth: keep get_current_user_for_workflow (session-or-API-key authN that does not hold a DB connection during the inline run, avoiding the SQLite lock contention api_key_security would cause) and enforce the base's RBAC on top: ensure_flow_permission(EXECUTE) before run, (READ) before status reconstruct, with widen_for_shares fetch. 2. Port the base's request-body globals onto the v2 WorkflowRunRequest. The X-LANGFLOW-GLOBAL-VAR-* headers stay supported (the Responses API passes globals that way); body globals win on conflict. Converters echo the effective globals via effective_globals. 3. Public endpoint keeps the v1 build_public_tmp posture (access_type==PUBLIC, run-as-owner); RBAC applies to the authenticated endpoint only. 4. Preserve the base's post-build KB-cache invalidation in the AG-UI build path. The endpoint, AG-UI bridge, pluggable stream adapters, public endpoint, and re-attach are unchanged. * feat(api): add output_text and session_id to v2 workflow response The synchronous /api/v2/workflows response keyed every result under its component id, so reading the answer meant knowing an id you can't predict. Surface two additive fields: - output_text: the flow's single text answer (ChatOutput/TextOutput). None when the flow has zero or multiple text outputs, so callers read outputs rather than the shortcut guessing which channel is the answer. - session_id: echoes the resolved session so chat/memory callers can continue the same thread (v1 /run returned this; v2 had dropped it). outputs is unchanged, so this is non-breaking. * test(api/v2): cover output_text and session_id on the v2 workflow response Pin the sync-response shortcuts on the v2 workflows endpoint: - output_text surfaces the lone ChatOutput/TextOutput text and stays None for non-output message nodes, data-only flows, and multi-text flows - session_id echoes the resolved session; the error response exposes neither - each outputs entry exposes only {type, status, content, metadata}, with the component id carried by the dict key Also drop the component_id kwarg the converter passed to ComponentOutput, which has no such field and silently dropped it. * feat(api/v2): structured output with resolution reason on v2 response Replace the flat output_text shortcut with an `output` object carrying the resolved text answer plus a `reason` that explains why it resolved that way (single/multiple/none/non_string/failed), so a null answer is always diagnosable instead of silently None. `reason` follows the LLM-domain finish_reason/stop_reason convention, distinct from the lifecycle status. Also add `display_name` to each ComponentOutput (the stable component id stays the dict key) and a computed `has_errors` flag derived from errors. * feat(api/v2): add request-side output selection (output_ids) Let a sync caller name the output(s) they want via output_ids so output.text resolves deterministically (reason=single) on multi-output flows instead of going null. Selection is steer-only: it picks the answer among the named outputs without filtering the outputs map. Invalid ids are rejected with 422 before the flow runs (and before any job row is created), so a typo costs no compute. Resolution considers selected outputs that actually fired, so branching flows resolve to whichever candidate ran. * feat(api/v2): emit per-output events on the langflow stream Give v2-workflows sync and the langflow stream protocol one parser. The stream now emits a normalized "output" event per terminal output carrying an OutputEvent (the ComponentOutput shape sync returns in outputs[id], plus component_id). A shared build_component_output() backs both the sync converter and the adapter, and the build loop ships authoritative vertex metadata as an additive output_meta key on end_vertex (existing consumers read build_data and ignore it). This is access-pattern parity (one parser, same fields, same terminal set), not byte-identical content: the stream reuses the v1 build path whose display serialization differs from sync's run_graph output. * feat: make content_blocks the source of truth for Message content Migrate Message.text from a Pydantic field to a @computed_field over content_blocks, and unify ContentBlock into the discriminated ContentType union so a Message's payload is one uniform shape. Schema changes: - Add 7 new content types (Image, Audio, Video, File, Reasoning, Usage, Citation) with validators for media sources, non-negative tokens, and ordered citation indices - Promote 'contents: list[ContentType]' to BaseContent so any node can nest (multimodal tool outputs, multi-step reasoning, grouped errors) - Fold ContentBlock into BaseContent and into the ContentType union with tag 'group'; content_blocks is now 'list[ContentType]' everywhere - Fix Data.__setattr__ to route through property descriptors via MRO walk Setter / serialization: - text setter appends a single TextContent at the end of content_blocks, preserving non-text blocks in chronological order (tool calls first, final text last) - model_post_init preserves explicit None in data['text'] when no TextContent exists in content_blocks, so callers can still distinguish 'text was never set' from 'text was set to empty string' from_lc_message: - Handle AIMessage tool_calls and usage_metadata regardless of whether content is a string or a list (tool-calling agents commonly emit content='' alongside tool_calls) - Tolerate explicit source=None in multimodal image payloads MessageResponse.from_message / MessageTable.from_message: - Accept any of (data['text'] set, text_stream pending, content_blocks non-empty) as 'content present', so tool-call-only and media-only messages persist rather than getting rejected as missing required fields Tests cover all new content types, the unified ContentType union, the text/content_blocks contract, the setter's chronological append, and the required-fields gate. * feat: stable id on content blocks + plumb LangChain tool_call_id Adds an optional 'id: str | None' field to BaseContent for stable identity across re-emissions of the same logical block. Producers that have a natural id (LangChain tool_call_id, external API id, a UUID stamped before the first emission) set it; consumers use it for dedup and cross-frame correlation. Without an id, consumers fall back to position-derived dedup, which assumes content_blocks is append-only within a message lifetime. Plumbs LangChain's 'tool_call_id' through 'Message.from_lc_message' into 'ToolContent.id'. The same logical tool call across start, args streaming, and result lifecycle now carries the same id, so a re-fired add_message dedups to one ToolContent instead of producing duplicates. Tests cover id default/round-trip/inheritance across every concrete content type, plus tool_call_id stability across repeated conversion, multiple tool calls each keeping their own id, and tool_calls alongside string content. * fix(schema): MessageResponse parses microsecond timestamps and ContentBlock partial updates preserve unset fields Two schema regressions surfaced in QA across the content-blocks chain: 1. MessageResponse.timestamp was typed as a bare datetime, but Message.timestamp default is a string with microsecond precision and a UTC timezone label ('%Y-%m-%d %H:%M:%S.%f %Z') that Pydantic's default datetime parser rejects. Any freshly built Message routed through MessageResponse.from_message raised ValidationError. Reuse the shared str_to_timestamp_validator so MessageResponse accepts every format Message itself recognises. 2. ContentBlock.__init__ marked every field as model_fields_set, not just the discriminator. The override defeated exclude_unset for the group content type: a patch like ContentBlock(title='new') dumped every defaulted field and, when merged onto an existing block by aupdate_messages, overwrote fields the caller never touched. Mark only 'type' (the discriminator) so partial updates carry the variant tag without clobbering the rest. Adds regression tests in test_message_content_blocks.py: from_message round-trips Message.timestamp without crashing, and ContentBlock exclude_unset stays narrow to the explicit fields plus the discriminator. * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * feat(playground): content blocks frontend renderer for v2 workflows * feat(frontend): add v2 (beta) tab to the API Access modal Add a v1 / v2 (beta) version toggle above the language tabs. The v2 tab emits Python, JavaScript, and cURL snippets for POST /api/v2/workflows. The two examples are framed by outcome, not by API jargon: "Get the full result" (the default single JSON response) and "Stream the result as it runs", each with a one-line plain-language description. The streaming snippets consume the default langflow protocol (switch on the event field; handle add_message, token, and end) rather than forcing the agui protocol. The API key is read from an env var, and a short response peek shows the shape a caller gets back. * feat(frontend): v2 examples read the answer from output.text * feat(agent): interleaved text + tool_use rendering and tabbed tool-output visualizer * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes * fix(api/v2): buffer parallel messages in the AG-UI translator instead of dropping them Parallel components stream tokens for different message ids interleaved. The translator tracked a single open message: the first foreign token closed the open message and tombstoned its id, so every later event for it was dropped and its remaining text never reached the client. Tokens for a message that cannot take the wire now buffer until the open message genuinely ends (its add_message finalizer), then flush in arrival order; complete messages landing mid-stream buffer the same way instead of interleaving a second START. end/error drain all buffers before the terminal event. The wire still carries at most one open text message, so the stream stays AG-UI-conformant. * [autofix.ci] apply automated fixes * fix(api/v2): gate AG-UI message finalization on non-partial state and purge removed buffers A partial add_message re-fire (the agent path emits these at tool start/end for a message it is still streaming) was treated as the finalizer: it closed and tombstoned the id, so the post-tool answer was dropped. Only a non-partial add_message finalizes now; state defaults to complete, so payloads without properties are unchanged. remove_message now purges a buffered message and tombstones its id, so text the backend retracted is not flushed to the client later. * fix(playground): scheme-guard content block URLs against unsafe schemes Lift safeUrl into a shared chatComponents/url.ts and route the file content block's download anchor through it, degrading unsafe-scheme URLs to a non-clickable label. Gate the image/audio/video src through the same helper so a data:text/html or attacker-chosen src is dropped rather than rendered. Matches the existing citation sanitization in SourcesStrip. Adds file-branch tests for an http URL and a javascript: URL. * refactor(playground): extract content-block layout + media renderers, drop favicon egress - Extract resolveContentBlockLayout from the duplicated shape-detection block in bot-message and chat-message into chat-messages/utils, with a direct unit test (legacy group, interleaved flat, text-only, divergent text, edit mode). - Split image/audio/video/file renderers out of ContentDisplay into MediaContentDisplay so the dispatcher stops growing per content type. - Replace the Google favicon fetch in SourcesStrip with a local Globe icon so cited source domains aren't leaked to a third party. - Trim WHAT comments to WHY, add typed test factories. * feat: make content_blocks the source of truth for Message content Migrate Message.text from a Pydantic field to a @computed_field over content_blocks, and unify ContentBlock into the discriminated ContentType union so a Message's payload is one uniform shape. Schema changes: - Add 7 new content types (Image, Audio, Video, File, Reasoning, Usage, Citation) with validators for media sources, non-negative tokens, and ordered citation indices - Promote 'contents: list[ContentType]' to BaseContent so any node can nest (multimodal tool outputs, multi-step reasoning, grouped errors) - Fold ContentBlock into BaseContent and into the ContentType union with tag 'group'; content_blocks is now 'list[ContentType]' everywhere - Fix Data.__setattr__ to route through property descriptors via MRO walk Setter / serialization: - text setter appends a single TextContent at the end of content_blocks, preserving non-text blocks in chronological order (tool calls first, final text last) - model_post_init preserves explicit None in data['text'] when no TextContent exists in content_blocks, so callers can still distinguish 'text was never set' from 'text was set to empty string' from_lc_message: - Handle AIMessage tool_calls and usage_metadata regardless of whether content is a string or a list (tool-calling agents commonly emit content='' alongside tool_calls) - Tolerate explicit source=None in multimodal image payloads MessageResponse.from_message / MessageTable.from_message: - Accept any of (data['text'] set, text_stream pending, content_blocks non-empty) as 'content present', so tool-call-only and media-only messages persist rather than getting rejected as missing required fields Tests cover all new content types, the unified ContentType union, the text/content_blocks contract, the setter's chronological append, and the required-fields gate. (cherry picked from commit 3b92500349420e2651a08587aa4e4292314ef8bf) * feat: stable id on content blocks + plumb LangChain tool_call_id Adds an optional 'id: str | None' field to BaseContent for stable identity across re-emissions of the same logical block. Producers that have a natural id (LangChain tool_call_id, external API id, a UUID stamped before the first emission) set it; consumers use it for dedup and cross-frame correlation. Without an id, consumers fall back to position-derived dedup, which assumes content_blocks is append-only within a message lifetime. Plumbs LangChain's 'tool_call_id' through 'Message.from_lc_message' into 'ToolContent.id'. The same logical tool call across start, args streaming, and result lifecycle now carries the same id, so a re-fired add_message dedups to one ToolContent instead of producing duplicates. Tests cover id default/round-trip/inheritance across every concrete content type, plus tool_call_id stability across repeated conversion, multiple tool calls each keeping their own id, and tool_calls alongside string content. (cherry picked from commit a3e8b40811dd72eb1650b299dccb20be7cd0c8cf) * fix(schema): MessageResponse parses microsecond timestamps and ContentBlock partial updates preserve unset fields Two schema regressions surfaced in QA across the content-blocks chain: 1. MessageResponse.timestamp was typed as a bare datetime, but Message.timestamp default is a string with microsecond precision and a UTC timezone label ('%Y-%m-%d %H:%M:%S.%f %Z') that Pydantic's default datetime parser rejects. Any freshly built Message routed through MessageResponse.from_message raised ValidationError. Reuse the shared str_to_timestamp_validator so MessageResponse accepts every format Message itself recognises. 2. ContentBlock.__init__ marked every field as model_fields_set, not just the discriminator. The override defeated exclude_unset for the group content type: a patch like ContentBlock(title='new') dumped every defaulted field and, when merged onto an existing block by aupdate_messages, overwrote fields the caller never touched. Mark only 'type' (the discriminator) so partial updates carry the variant tag without clobbering the rest. Adds regression tests in test_message_content_blocks.py: from_message round-trips Message.timestamp without crashing, and ContentBlock exclude_unset stays narrow to the explicit fields plus the discriminator. (cherry picked from commit 6f6639374faf52bfd7d0041a3fa3aec7a1c26a8a) * fix(schema): address content_blocks review feedback - sync langflow-base ContentBlock.__init__ with the lfx copy (model_fields_set parity) so exclude_unset no longer clobbers type; add cross-module regression test - route MessageResponse content_blocks discriminator-first so stored flat blocks with contents=[] validate instead of raising - move Message SecretStr coercion into model_post_init and drop the dead validate_text before-validator - drop the no-op _fold_text_into_content_blocks validator - log a shape-only debug line when from_lc_message drops an undecodable image - type MessageResponse.content_blocks as list[ContentType] | None (cherry picked from commit 6c7cec8a529c38d2f9eb7a35f94277611c8696cc) * fix(agents): stop duplicating the final answer in content_blocks With content_blocks as the source of truth, Message.text is a computed field whose setter appends a trailing top-level TextContent. handle_on_chain_end also appended the same answer into the Agent Steps group, so the final answer rendered twice (assert 2 == 1 in test_multiple_events). The streaming path already relies on the setter alone; make the non-streaming path match. * chore: auto-bake note keys and regenerate backend locales/en.json [skip ci] * feat(schema): project new content_blocks back to the v1 wire shape The in-memory Message and the v2 (AG-UI) path use the new content_blocks union (groups tagged "group", every node carries id/contents, the agent answer is a trailing top-level TextContent). The v1 API keeps emitting the pre-1.11.0 shape via a pure legacy_render projection that runs only at the v1 boundaries: the v1 read/response models, the memories endpoint, the v1 build SSE stream, the webhook events SSE stream, and the /run response and stream. The build, webhook, and /run projections recurse so the Data mirror (data.data.content_blocks) is projected alongside the top-level copy. v2 serializes the live Message and keeps the new shape. * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * [autofix.ci] apply automated fixes * fix(api): keep simple_run_flow returning RunResponse, project v1 at the HTTP boundary simple_run_flow is a shared helper, so wrapping its return in a JSONResponse to apply the v1 content_blocks projection broke internal callers that call .model_dump() on the result (the streaming run_flow_generator and get_build_results). Return the RunResponse object from the helper and apply the projection at the non-stream HTTP boundary in _run_flow_internal instead. The streaming path already projects the end event via _project_run_event. * [autofix.ci] apply automated fixes * fix(frontend): route media content URLs through safeUrl The media content type rendered <img src={url}> with the raw url, unlike the sibling image/audio/video/file cases that guard via safeUrl. A javascript:/data: scheme from untrusted tool output reached the img src. Apply the same safeUrl guard and skip rendering when it returns null. * fix(frontend): detect untyped legacy groups in content-block layout resolveContentBlockLayout detected groups with type === "group", missing the legacy / v1-projected "Agent Steps" group that is persisted without a type field (just title + contents). Those untyped groups failed hasGroup and were miscounted as flat non-text items, wrongly enabling ordering mode so the duplicate top-level text rendered above the tools and the bubble body was suppressed. Use the shared isGroupedBlock predicate, matching the rest of the render pipeline. Also document the latent tool-less-group gap in ContentBlockDisplay (a no-tool answer projects to a text-only group; the drop is benign today but the gate keys off toolItems, not groupedBlocks). * fix(frontend): render a group's displayable non-tool content ContentBlockDisplay gated entirely on a group's tool_use leaves, so a group whose contents had no tool_use (reasoning / citation / media, …) rendered nothing. Collect a group's displayable non-tool leaves and render them through the same loose renderer as top-level flat leaves, while keeping text and usage out so the legacy v1 Input/Output scaffolding stays hidden (the bubble body already paints the answer). Adds collectGroupLooseLeaves and unit tests. * test(frontend): deep probes for ContentBlockDisplay group rendering Renders the real ContentBlockDisplay (real ContentDisplay / ToolCallCard / SourcesStrip / accordion; only ESM infra is mocked) and asserts the DOM for each content_blocks shape: a tool-less group's citation and media now render, an untyped legacy group renders its non-tool content, a tool-bearing group renders both the tool and its extra leaf, the legacy Input/Output text stays hidden, text-only and usage-only groups render nothing, and the flat shape is unchanged. Confirmed RED on the pre-fix component (4/8 fail) and GREEN after. * chore: auto-bake note keys and regenerate backend locales/en.json [skip ci] * chore: re-trigger CI after [skip ci] bake commit * fix: regenerate component_index.json after agent.py content_blocks change The merged index still carried AgentComponent's pre-#13390 code, so the runtime hash allow-list (built from the index) didn't match the live agent.py. That failed the custom-component admin-only known-template carve-out (403 instead of 200) and drifted Update Component Index. Rebuilt via 'make build_component_index'; sha256 now 070077b2. * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * feat(api): durable background execution service (store + default backend) (#13507) * feat: native v2 workflows endpoint with pluggable stream protocols Rebased onto release-1.10.0. The base independently rebuilt the v2 workflows backend (RBAC, body globals, share-aware fetch); keep our forward design and conform its auth to that work: 1. Auth: keep get_current_user_for_workflow (session-or-API-key authN that does not hold a DB connection during the inline run, avoiding the SQLite lock contention api_key_security would cause) and enforce the base's RBAC on top: ensure_flow_permission(EXECUTE) before run, (READ) before status reconstruct, with widen_for_shares fetch. 2. Port the base's request-body globals onto the v2 WorkflowRunRequest. The X-LANGFLOW-GLOBAL-VAR-* headers stay supported (the Responses API passes globals that way); body globals win on conflict. Converters echo the effective globals via effective_globals. 3. Public endpoint keeps the v1 build_public_tmp posture (access_type==PUBLIC, run-as-owner); RBAC applies to the authenticated endpoint only. 4. Preserve the base's post-build KB-cache invalidation in the AG-UI build path. The endpoint, AG-UI bridge, pluggable stream adapters, public endpoint, and re-attach are unchanged. * feat(api): add output_text and session_id to v2 workflow response The synchronous /api/v2/workflows response keyed every result under its component id, so reading the answer meant knowing an id you can't predict. Surface two additive fields: - output_text: the flow's single text answer (ChatOutput/TextOutput). None when the flow has zero or multiple text outputs, so callers read outputs rather than the shortcut guessing which channel is the answer. - session_id: echoes the resolved session so chat/memory callers can continue the same thread (v1 /run returned this; v2 had dropped it). outputs is unchanged, so this is non-breaking. * test(api/v2): cover output_text and session_id on the v2 workflow response Pin the sync-response shortcuts on the v2 workflows endpoint: - output_text surfaces the lone ChatOutput/TextOutput text and stays None for non-output message nodes, data-only flows, and multi-text flows - session_id echoes the resolved session; the error response exposes neither - each outputs entry exposes only {type, status, content, metadata}, with the component id carried by the dict key Also drop the component_id kwarg the converter passed to ComponentOutput, which has no such field and silently dropped it. * feat(api/v2): structured output with resolution reason on v2 response Replace the flat output_text shortcut with an `output` object carrying the resolved text answer plus a `reason` that explains why it resolved that way (single/multiple/none/non_string/failed), so a null answer is always diagnosable instead of silently None. `reason` follows the LLM-domain finish_reason/stop_reason convention, distinct from the lifecycle status. Also add `display_name` to each ComponentOutput (the stable component id stays the dict key) and a computed `has_errors` flag derived from errors. * feat(api/v2): add request-side output selection (output_ids) Let a sync caller name the output(s) they want via output_ids so output.text resolves deterministically (reason=single) on multi-output flows instead of going null. Selection is steer-only: it picks the answer among the named outputs without filtering the outputs map. Invalid ids are rejected with 422 before the flow runs (and before any job row is created), so a typo costs no compute. Resolution considers selected outputs that actually fired, so branching flows resolve to whichever candidate ran. * feat(api/v2): emit per-output events on the langflow stream Give v2-workflows sync and the langflow stream protocol one parser. The stream now emits a normalized "output" event per terminal output carrying an OutputEvent (the ComponentOutput shape sync returns in outputs[id], plus component_id). A shared build_component_output() backs both the sync converter and the adapter, and the build loop ships authoritative vertex metadata as an additive output_meta key on end_vertex (existing consumers read build_data and ignore it). This is access-pattern parity (one parser, same fields, same terminal set), not byte-identical content: the stream reuses the v1 build path whose display serialization differs from sync's run_graph output. * feat(api/v2): durable background execution service (store + default backend) Turns v2 mode:background into a durable, in-API background execution service behind a BackgroundExecutionService facade. Adds the store layer (result/error columns, job_events durable milestone log, execution_signals control, heartbeat/lease, 3 migrations), the default backend (bounded executor, runner, in-memory live bus, liveness-aware single-flight orphan sweep), the v2 endpoint rewiring, and the real-instance test harness. Needs no new infra; works on the SQLite single-process install. The redis-scaled worker backend is stacked on top in a follow-up PR. * test(background-execution): rename hard_proof marker to real_services The hard_proof marker name was a vibe word that said nothing about what the tests need. Rename it to real_services everywhere: the pytest marker registration, the *_hard_proof.py test files, the Makefile target (real_services_tests), the -m selector in migration-validation.yml, and the CI job. real_services says what these tests require: real Postgres + Redis + worker subprocesses. (integration was already taken for the external-API suite under tests/integration.) * fix(api/v2): enforce no-code-execution gate on public workflow endpoint The v2 public endpoint only ran validate_flow_for_current_settings and skipped validate_public_flow_no_code_execution, which the v1 build_public_tmp path applies. A public flow containing a Python interpreter/REPL (or the legacy Python Code Structured tool, Smart Transform lambda) was therefore an unauthenticated server-side code-execution primitive (report H1-3754930). Mirror v1: import the validator and call it right after the public-access gate. PublicFlowValidationError subclasses CustomComponentValidationError, so the existing handler already sanitizes it to a 400 'This flow cannot be executed.' without leaking the blocked component class names. Add a non-mocking test that builds a public flow with a real PythonREPLComponent and asserts the sanitized 400 (verified RED: returns 200 without the gate). LE-1389 * fix(api/v2): reconstruct background workflow status from job-keyed vertex builds A completed background job's GET status 500'd with 'No vertex builds found for job_id'. The background build path differed from the sync path twice: 1. generate_flow_events minted a fresh run_id instead of using job_id, so vertex builds were keyed by an id the status query never uses. Thread run_id through _stream_event_frames -> generate_flow_events and pass job_id from the background buffer so graph.run_id == job_id (the sync path already does graph.set_run_id(job_id)). 2. The SSE build loop (build_vertices) only persisted builds when log_builds was set and never passed job_id. Tie log_builds to job-tracked runs (run_id present) and pass job_id=graph.run_id on the persist call. Job-tracked runs also persist streaming terminal vertices so reconstruction is complete; the live build path (run_id is None) keeps its original behavior, so the v1 build path is unchanged. Test: a real background run polled to completion, then GET status asserts a reconstructed 200 (verified RED: 500 'No vertex builds found' before the fix). Covers the non-streaming flow. v1 build path unchanged (35 build tests pass); AG-UI suite 46 pass. LE-1389 * fix(api/v2): merge workflow AG-UI cancellation hardening LE-1389 * fix(api/v2): signal cross-worker workflow stops LE-1389 * fix(api/v2): report unconfirmed workflow stops LE-1389 * fix(api/v2): keep background workflows out of polling watchdog LE-1389 * fix(api/v2): buffer parallel messages in the AG-UI translator instead of dropping them Parallel components stream tokens for different message ids interleaved. The translator tracked a single open message: the first foreign token closed the open message and tombstoned its id, so every later event for it was dropped and its remaining text never reached the client. Tokens for a message that cannot take the wire now buffer until the open message genuinely ends (its add_message finalizer), then flush in arrival order; complete messages landing mid-stream buffer the same way instead of interleaving a second START. end/error drain all buffers before the terminal event. The wire still carries at most one open text message, so the stream stays AG-UI-conformant. * fix(api/v2): gate AG-UI message finalization on non-partial state and purge removed buffers A partial add_message re-fire (the agent path emits these at tool start/end for a message it is still streaming) was treated as the finalizer: it closed and tombstoned the id, so the post-tool answer was dropped. Only a non-partial add_message finalizes now; state defaults to complete, so payloads without properties are unchanged. remove_message now purges a buffered message and tombstones its id, so text the backend retracted is not flushed to the client later. * Fix AG-UI workflow lifecycle edges * [autofix.ci] apply automated fixes * fix(frontend): enable downlevelIteration for jest Set/Map iteration ts-jest compiles with target es5; without downlevelIteration, [...set] and for...of over a Set/Map emit ES5 that yields nothing. That silently broke the AG-UI bridge tests: runningNodeIds spread, markRunningNodesFailed, and restoreOriginalBuildStatuses all iterated empty. Production (Vite/SWC, modern target) was never affected; only the ts-jest harness was. Fixes the 3 failing jest tests on this branch with no other suite changes (4994/4994 pass). * fix(api/v2): surface inactivated branch vertices over AG-UI A branch component (If-Else, Conditional Router) reports its not-taken vertices in build_data.inactivated_vertices, but the AG-UI translator only emitted the branch node's own success/error status and dropped that list. The canvas seeds every planned node as pending from vertices_sorted; skipped vertices then get no build_start/end_vertex, so they stayed stuck on pending instead of rendering as inactive (the v1 build path marked them INACTIVE). The translator now appends an inactive STATE_DELTA op per inactivated vertex, and the frontend bridge maps the new inactive status to BuildStatus.INACTIVE and tears its edges down like a completed node. Fixes the If-Else regression in general-bugs-reset-flow-run.spec.ts. * fix(api/v2): dedupe repeated inactive node deltas in AG-UI stream build.py keeps reporting a conditionally-excluded vertex in inactivated_vertices on every subsequent end_vertex (the excluded set persists until the ConditionalRouter clears it), so the translator was putting the same inactive STATE_DELTA on the wire once per remaining vertex. Track emitted inactive nodes and skip re-emitting; drop a node from the set when it actually runs again (build_start/end_vertex) so a loop re-activation can still re-emit inactive later. * fix(api/v2): no duplicate WORKFLOW job row on durable background runs A v2 background run created TWO JobType.WORKFLOW rows for one flow execution: the durable row (submit()'s job_id, owned by JobRunner) plus an orphan keyed by the run_id generate_flow_events mints, because the build pipeline's track_job_status defaults True and the durable frame source never passed False. The flow ran once (double bookkeeping), but every background run left a phantom WORKFLOW row + job_events and double-fired the memory-base hook, skewing metrics. Thread track_job_status through _stream_event_frames; pass False only from the background frame source (the durable runner already owns the row + fires the hook with the durable job_id). Stream/public paths keep default True. Also gate build.py's memory-base hook fire behind track_job_status so background doesn't double-fire. Adds a regression test (RED before fix: found 2 rows). * test(api/v2): update stale workflow-stop tests for the durable design These 3 tests targeted the removed queue-service stop helper (_cancel_workflow_queue_job / get_queue_service), inherited via the agui->bg-default merge and failing with AttributeError across the stack: - test_stop_workflow_success: adapted to the durable stop path (revoke_task -> stop_job -> update_job_status(CANCELLED)). - test_stop_workflow_allowed_for_legacy_job_with_no_user_id (IDOR): adapted to the durable mechanism; still asserts the ownership check does not block a legacy user_id=None row. - test_stop_workflow_returns_503_when_queue_cancel_cannot_be_confirmed: dropped — the durable stop writes a best-effort STOP signal and always finalizes CANCELLED; the queue-service 'cannot confirm -> 503' path no longer exists. test_workflow.py now passes 24/24. * fix(api/v2): move FrameSourceFactory alias under TYPE_CHECKING As a module-level runtime value, FrameSourceFactory = Callable[..., Any] is a GenericAlias that passes isinstance(obj, type) but makes issubclass(obj, Service) raise on Python 3.10/3.14. The service factory scans this module for Service subclasses (services/factory.py:90), so the runtime alias crashed service initialization on those interpreters with 'issubclass() arg 1 must be a class' -> 'Could not initialize services', erroring out dozens of unrelated tests at setup (3.13 was unaffected, which is why local runs passed). The alias is only referenced in a lazy annotation (from __future__ import annotations), so moving it + the Callable import under TYPE_CHECKING removes it from the runtime namespace with no behavior change. Verified: alias absent from runtime module namespace, factory scan finds BackgroundExecutionService cleanly, durable service tests 26/26 pass. * test(lfx): register background-execution Settings fields in the field-count gate The durable background-execution work added six Settings fields (background_max_concurrency, background_job_timeout, background_lease_ttl_s, background_heartbeat_interval_s, background_watchdog_interval_s, test_redis_url) without updating EXPECTED_FIELDS, so test_field_count_unchanged failed 152 != 146. These are intentional bg-exec config; add them to the gate. * fix(api/v2): restore "end" side-channel event in AG-UI workflow stream The durable background-execution rewrite of workflow.py reverted `side_channel_events` to its pre-"end" form, dropping the "end" event from the AG-UI side-channel. That event carries `build_duration` to the playground chat-view, and the message metadata badge only renders when `hasDuration || hasTokens`. With build_duration gone the badge vanished, failing the token-usage and shareable-playground "Finished In" regression tests. Re-add "end" so the streaming playground path delivers it again. * fix(api/v2): apply request tweaks on the streaming and background paths The v2 workflows endpoint applied `tweaks` only on mode=sync. The stream and background paths build the graph via the v1 build-vertex loop (`generate_flow_events`), which never received the tweaks, so they were silently dropped. The confusing symptom: a model passed via tweaks surfaced as "A model selection is required", and any per-component override was ignored on non-sync runs. Thread `parsed.tweaks` into `generate_flow_events` and apply them to the built graph via `vertex.update_raw_params`. We do not use the lfx `process_tweaks_on_graph` helper because it only sets `vertex.params`, which does not persist to runtime (the same bug `lfx.base.tools.run_flow._process_tweaks_on_graph` works around). No-tweaks runs are unchanged (guarded by `if tweaks`). Adds a streaming regression test that overrides ChatInput via tweaks and asserts the value drives the run. * fix(api/v2): return background run output from completed status A completed background run's GET status returned a bare COMPLETED with an empty `outputs` and a null `output`. The COMPLETED branch reconstructs from `vertex_builds` keyed by job_id, which the durable path does not write, so reconstruction raised ValueError and fell through to an empty response; the result was only retrievable via a /events re-attach. The runner now captures the terminal `output` events (the langflow adapter's normalized ComponentOutput payloads) into `Job.result`, and the status COMPLETED branch rebuilds the `outputs` map and resolved `output` from them via `workflow_response_from_output_events`, matching the sync response. agui-protocol runs emit no `output` events, so their status stays result-less (the result remains on the /events log). * fix(api/v2): address review findings on the v2 workflows endpoint - recover session_id for completed background jobs from the persisted terminal message instead of always returning null, so GET status can continue the same chat/memory thread - replay a user-cancel as a CUSTOM cancel marker + RUN_FINISHED (agui) and a `cancelled` terminal (langflow) instead of RUN_ERROR, so a re-attaching client no longer reads a deliberate stop as a failure - cancel the evicted still-running buffer writer when the background-run registry is full, so it stops appending into a run no reader can find - derive per-component status from the error artifact / valid flag instead of hardcoding COMPLETED, and stop the langflow adapter dropping `valid` - throttle the unauthenticated public endpoint per IP and bound its input_value/session_id length - document the sync-only scope of request-body globals - document that live event re-attach is intentionally owner-only * test(lfx): register public_flow_rate_limit_per_minute in settings composition * refactor(v2 workflows): split workflow.py and address review blockers Splits the ~1.5k-line workflow.py into focused modules and folds in the execution-timeout and error-sanitization fixes from Cristhianzl's review of #13307. - B1: workflow.py now holds only the four route handlers. Validation guards move to workflow_validation, the sync/stream run loop to workflow_execution, and the durable background machinery to workflow_background (layered, acyclic). - I1: add workflow_execution_timeout (default 300) and apply a single wall-clock ceiling across sync, stream, background, and public via _stream_event_frames. A timeout becomes a sanitized terminal error and marks a background job failed. - I3: the route error handlers no longer echo raw exception text. They return a generic, code-tagged message and log the full exception server-side. - R1: remove the "commented out / future scope" comments that sat over live dataframe-extraction code in converters.py. - R4: drop the worker-routing internals from the reattach 409 message. Tests cover the timeout terminal-error path and the error-body sanitization, and the settings field-count guard is updated for the new setting. * refactor(lfx): extract v2 workflow contract layer into lfx.workflow Moves the protocol-agnostic pieces of the v2 workflows API out of the langflow backend into lfx so both the backend and `lfx serve` can share one contract. First step toward giving lfx (the production runtime) the v2 workflows API. - Move api/v2/adapters/, agui_translator.py, and converters.py to lfx/workflow/. They depend only on lfx.schema.workflow and ag_ui (already an lfx dep), so lfx carries the contract with zero langflow imports. - Decouple the one langflow reference: converters typed run_response against langflow.api.v1.schemas.RunResponse (TYPE_CHECKING only). Replaced with a local RunResponseLike Protocol (outputs + session_id), the only attributes used. - Repoint the six backend v2 workflow modules to import from lfx.workflow. - Move the five protocol-agnostic contract tests into src/lfx/tests/unit/workflow/ (run in the lfx-only env). test_output_event_parity and test_workflow_agui stay in langflow (they need langflow.api.build) with repointed imports. Coverage unchanged: 201 contract tests pass in the lfx-only env, 191 backend v2 tests pass; 392 total, same as before the move. * fix(background-execution): prevent worker deadlock on stop() under Python 3.10 The bounded executor's worker awaited the in-flight job task with a bare `await task`. On Python 3.10, when stop() cancels a worker while its job task is finishing, the awaiter's wakeup is lost and the event loop idles forever in select(), deadlocking stop(). Await via a done-callback Event (the same mechanism stop()'s own asyncio.gather already uses), which delivers the wakeup reliably; task.result() preserves the cancellation and exception semantics of the bare await. * fix(background-execution): keep ephemeral frames on reattach, pin psycopg in real-service tests The real-service CI job broke on `ModuleNotFoundError: No module named 'asyncpg'`. asyncpg was never declared anywhere in this repo; it arrived transitively through `cuga`, which release-1.11.0 dropped when it moved the root dep to `lfx-bundles[all-no-torch]` (#13886). Normalize the harness URL to `postgresql+psycopg` instead: it is what `--extra postgresql` installs and what DatabaseService already selects for async Postgres, so the test now exercises the production driver rather than a stowaway. Also fix a real reattach bug. The runner publishes ephemeral token frames tagged with the last durable seq (they have no job_events row), while reattach's tail skipped anything with `seq <= highest`. After replay left `highest` at that same seq, every token delta was dropped until the next durable milestone advanced it, so a reconnect mid-stream saw no tokens. Mark frames durable/ephemeral and dedupe only the durable ones, which are the only frames a replay can return. Log instead of silently suppressing a failed stop_job signal, and drop a comment block duplicated verbatim in InProcessExecutor.stop(). * fix(migrations): merge alembic heads (mcp_server + execution_signals) --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * docs: AGUI protocol, workflows quickstart, update examples (#13911) * docs: curl code snippet contains placeholder * docs: update workflows example to match new schema * docs: include ag-ui in workflow api reference * docs: workflows api agui streaming * docs: workflows api quickstart and ag-ui example * docs: fix broken build links * docs: agui peer review * feat(frontend): render the flow share seam in the editor Share dropdown (#14009) The CustomFlowShareAction customization seam only rendered in the flow list's card menu, so overlays that implement user/team sharing had no entry point inside the flow editor. Render the same seam at the top of the editor's Share dropdown; the OSS stub renders nothing, so the OSS menu is unchanged. Adds an optional menuContext prop to the seam so overlays can label the editor placement differently from the card menu. * fix: fail loud on undecryptable KB embedding key with recovery path (#13806) * fix: fail loud on undecryptable KB embedding key with recovery path - Add KBKeyDecryptError exception for SECRET_KEY rotation scenarios - Implement require_api_key flag in load_kb_metadata for critical vs non-critical paths - Retrieval: raise when component doesn't supply key, allow component key bypass - Ingestion: raise when no component key, warn when component key used as fallback - Add comprehensive test coverage for decrypt failure scenarios - Consolidate _kb_paths imports in knowledge.py * chore: auto-bake note keys and regenerate backend locales/en.json [skip ci] * chore: auto-bake note keys and regenerate backend locales/en.json [skip ci] * chore: trigger CI * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) --------- Co-authored-by: Janardan S Kavia <janardanskavia@Janardans-MacBook-Pro.local> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Eric Hare <ericrhare@gmail.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * feat(models): add Azure AI Foundry to unified model provider setup (#13912) * feat(models): add Azure AI Foundry to unified model provider setup Expose Azure AI…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #13307. First of a two-PR stack. This one turns the v2
mode: backgroundstub into an actual durable background execution service that needs no new infrastructure (it works on the default SQLite single-process install). The redis-scaled worker backend is stacked on top in a follow-up PR.What it does
The old
mode: backgroundran the flow as a fire-and-forgetasyncio.create_taskin the API worker, buffered SSE frames in a process-local dict, and persisted only job metadata. A restart killed the run, reattach/stop only worked on the worker that served the POST, and a completed job's output vanished. This replaces that with oneBackgroundExecutionServicefacade over small primitives (store / queue / runner / live bus / control), wired here to the in-process default backend:execution_signalsDB row, and a liveness-aware single-flight startup sweep.Same
WorkflowJobResponseand SSE /Last-Event-IDcontract. The change is additive: a client that works now keeps working, it just stops losing jobs.Data model (three migrations)
result/errorcolumns onjob, a newjob_eventstable (the durable milestone log,UNIQUE(job_id, seq),seqis theLast-Event-IDcursor), and a newexecution_signalstable (cross-worker stop without redis). No newJobStatusvalues. A heartbeat/lease lives injob_metadataso the sweep only ever reconciles genuinely orphaned runs.Semantics
At-least-once for not-yet-started (QUEUED) work, at-most-once for in-flight work by default (a crashed run becomes FAILED, not silently re-run, since flows are not generally idempotent), and opt-in bounded retry per flow. Durable result/error plus a terminal
job_eventsrow on every terminal path including TIMED_OUT and CANCELLED. Cooperative stop at vertex boundaries. Cross-tenant safe: the idempotency key is scoped per user. Inline requestglobalsare redacted from the persisted re-enqueue request (use named stored variables for secrets in background runs).What's proven
Built TDD against real Postgres (and SQLite), no fakes on the durability paths. The default-backend crown jewels are covered: restart-survival (a fresh service instance against the same DB replays the durable milestones and returns, a hang fails the test), the orphan sweep only fails genuinely-orphaned runs (a heartbeating job on another replica is spared), bounded concurrency, and the
Last-Event-IDreattach contract (live and durable-replay frames share one seq namespace, no gap or duplicate). The test harness also runs the durability tier against real Postgres in CI.There were several rounds of review while building this; the findings (a restart reattach hang, a
Last-Event-IDseq namespace mismatch, a multi-worker orphan-sweep race, a per-user idempotency leak, an attempt-cap race) are all fixed with regression tests in this PR.Notes
HITL is deliberately deferred, but the lifecycle (extensible
JobStatus) and the seq-based event log are shaped so a later resume-from-checkpoint can drop in.Summary by CodeRabbit
Release Notes
New Features
Tests