Skip to content

QA-CHECKLIST.md

Latest commit

 

History

History
790 lines (641 loc) · 38.7 KB

QA-CHECKLIST.md

File metadata and controls

790 lines (641 loc) · 38.7 KB
Raw

Langflow — Regression Test Checklist

Repository: C:/QAx/langflow-playwright/langflow-e2e Tests: tests/tests-automations/regression/ Config: playwright.config.ts Last updated: 2026-05-05

How to use this checklist

  • [x] → automated and validated — only assigned when the underlying Playwright test carries the @stable tag
  • [-] → automated, needs validation (test exists but is not yet @stable, or the entry refers to a Page/Helper rather than a test)
  • [ ] (empty) → needs automation
  • [~] → partially covered
  • [!] → covered but flaky / unstable

PART I — PAGES & HELPERS

Pages

  • [-] SimpleAgentTemplatePage — loads Simple Agent template with configurable provider and model → pages/SimpleAgentTemplatePage.ts
  • [-] SettingsPage — navigation to the settings page via user menu → pages/SettingsPage.ts
  • Component sidebar — component navigation bar with searchable parameter support
  • Model Provider — navigation to the model provider management tab
  • API Keys — navigation to the API keys / global variables tab
  • Templates — navigation to the template selection tab (Starter Projects)
  • Import Flow — navigation to import a flow via JSON
  • Delete Flow — navigation to delete a flow
  • MCP Config — navigation to configure MCP Server

Helpers

Provider Setup

  • [-] OpenAI Provider Setup → helpers/provider-setup/setup-openai.ts
  • [-] Anthropic Provider Setup → helpers/provider-setup/setup-anthropic.ts
  • [-] Google Generative AI Provider Setup → helpers/provider-setup/setup-google.ts
  • [-] Provider Map (providerSetupMap) — central registration point → helpers/provider-setup/index.ts
  • [-] Provider validation via API (credit, valid key) → helpers/provider-setup/collect-models.ts
  • [-] Collection of available models via UI (Settings → Model Providers) → helpers/provider-setup/collect-models.ts
  • [-] providers.json — status of each provider (active/inactive + reason) → data/providers.json
  • [-] models.json — list of models per provider → data/models.json

Flows

  • [-] Load Simple Agent with variable provider and model → pages/SimpleAgentTemplatePage.ts
  • [-] Load Simple Agent with OpenAI (wrapper) → helpers/flows/load-simple-agent-with-openai.ts

To implement

  • Configure an MCP
  • Configure a Custom Component
  • Delete a component
  • Run a flow
  • Pause a flow
  • Send a chat input
  • Verify the chat output

PART II — TEST AUTOMATION COVERAGE

Organized according to tests/tests-automations/regression/

api/ — REST API

api/flows/ — REST API

1.1 Health Check

  • [-] GET /api/v1/health_check → status 200, db ok
  • [-] GET /api/v1/health → returns uptime and version

1.2 Flow CRUD via API

  • [-] POST /api/v1/flows/ → creates flow, returns ID
  • [-] GET /api/v1/flows/ → lists user flows
  • [-] GET /api/v1/flows/{id} → returns flow by ID
  • [-] PATCH /api/v1/flows/{id} → updates name/description
  • [-] DELETE /api/v1/flows/{id} → removes flow, returns 200
  • [-] GET /api/v1/flows/{id} after DELETE → should return 404

1.3 Flow Execution via API

  • [-] POST /api/v1/run/{flow_id} with input_value → returns response
  • [-] POST with tweaks → parameters override flow configuration
  • [-] POST with custom session_id
  • [-] POST with input_type: "chat" and output_type: "chat"
  • POST with invalid API key → returns 401/403 → api-invalid-key.spec.ts
  • [-] POST to non-existent flow → returns 404

1.4 Components via API

  • [-] GET /api/v1/all → lists all available components
  • [-] POST /api/v1/custom_component → creates custom component

1.5 Messages and Monitoring via API

  • [-] GET /api/v1/monitor/messages → returns 200 with array
  • [-] GET with session_id filter returns only messages from that session

1.6 Integration Code Generation

  • [-] Generate curl for API execution
  • [-] Generate Python code for integration
  • [-] API access modal

core-components/ — Component Configuration + Core Components

2. Component Configuration

2.1 Parameters Panel

  • [-] Open component advanced options
  • [-] Edit text field (input)
  • [-] Edit dropdown
  • [-] Edit text area (textarea)
  • [-] Edit code field
  • [-] Edit float field
  • [-] Edit int field
  • [-] Edit toggle field
  • [-] Edit key-pair list
  • [-] Edit input list
  • [-] Edit table input
  • [-] Edit slider
  • [-] Edit tab component

2.2 Tool Mode

  • [-] Enable Tool Mode on a component
  • [-] Group components in Tool Mode
  • [-] Edit tools (edit-tools)

2.3 Component Updates

  • [-] Outdated component notification
  • [-] Update component action
  • Update with breaking change — should alert user
  • Legacy component visible via configuration

2.4 Code Editing

  • [-] Edit Python code of custom component
  • [-] Full custom component

3. Core Components

3.1 Chat Input / Output

  • [-] ChatInput receives user message
  • [-] ChatOutput displays LLM response
  • [-] Chat Input/Output with user authentication

3.2 Prompt Template

  • [-] Prompt with variables in curly braces
  • [-] Prompt modal
  • [-] Dynamic port generated when adding a variable to the prompt
  • [-] Removing a variable from the prompt deletes the corresponding port

3.3 API Request (HTTP)

  • Renders on canvas with URL and API Response handles → core-components/api-request-component-regression.spec.ts
  • Inspector fields accept URL and HTTP method values → core-components/api-request-component-regression.spec.ts
  • Execute GET request and verify 200 status and output structure → core-components/api-request-component-regression.spec.ts
  • Execute POST request and verify POST verb is sent (status 200) → core-components/api-request-component-regression.spec.ts
  • Execute PUT request and verify PUT verb is sent (status 200) → core-components/api-request-component-regression.spec.ts
  • Execute PATCH request and verify PATCH verb is sent (status 200) → core-components/api-request-component-regression.spec.ts
  • Execute DELETE request and verify DELETE verb is sent (status 200) → core-components/api-request-component-regression.spec.ts
  • Non-2xx HTTP response (404) propagated as status_code without crash → core-components/api-request-component-regression.spec.ts
  • Query parameters embedded in URL are sent and echoed in response → core-components/api-request-component-regression.spec.ts
  • Invalid URL error shows notification with descriptive error message → core-components/api-request-component-regression.spec.ts
  • Headers table accepts key + value cell entries via inspector → core-components/api-request-component-regression.spec.ts
  • cURL tab switches mode and exposes the cURL input field → core-components/api-request-component-regression.spec.ts
  • cURL parser auto-fills URL field and executes the GET, returning 200 → core-components/api-request-component-regression.spec.ts
  • Body table key + value entries (body field is advanced=True)
  • Flow state persisted in database after autosave

3.4 Webhook

  • POST aceita JSON e text/plain retornando 202 com status: "in progress" → core-components/webhook-component-regression.spec.ts
  • Flow salvo no banco contém o nó Webhook com endpoint="BACKEND_URL" → core-components/webhook-component-regression.spec.ts
  • Campo cURL no inspector mostra URL válida com flow ID e flags corretas (-X POST, Content-Type, -d) → core-components/webhook-component-regression.spec.ts
  • Data field vazia retorna objeto Data vazio {} ao executar → core-components/webhook-component-regression.spec.ts
  • Campo endpoint (str_endpoint) renderiza a URL real do webhook → core-components/webhook-component-regression.spec.ts
  • Botão de cópia copia a URL correta para o clipboard e exibe toast "Endpoint URL copied" → core-components/webhook-component-regression.spec.ts
  • POST para flow inexistente retorna 404 → core-components/webhook-component-regression.spec.ts
  • GET /api/v1/monitor/messages retorna 200 com array → core-components/webhook-component-regression.spec.ts
  • Payload JSON recebido é propagado corretamente como saída Data do componente → core-components/webhook-component-regression.spec.ts
  • Payload inválido (não-JSON) é encapsulado em {"payload": "..."} na saída → core-components/webhook-component-regression.spec.ts

3.5 Agent (Component)

  • [-] Agent component displayed on canvas with default settings
  • Configure system prompt in Agent component → agent-system-prompt.spec.ts
  • Configure model provider directly in Agent component → agent-provider-field-isolation.spec.ts

3.6 Loop Component

  • Loop component renders on canvas with title and run button → core-components/loop-component-regression.spec.ts
  • Correct handles: inputs-left, item-left, item-right, done-right → core-components/loop-component-regression.spec.ts
  • Output inspection buttons present for item and done → core-components/loop-component-regression.spec.ts
  • Run without connections shows "Flow build failed" notification without crash → core-components/loop-component-regression.spec.ts
  • Loop iterates over 2 ArXiv articles (Research Translation Loop template) and aggregates response in Playground → core-components/loop-component-regression.spec.ts
  • Loop stops when exit condition is met

3.7 Nested / Grouping

  • [-] Nested component
  • [-] Enter and exit grouped component

core-functionality/ — Core and Operational Logic

core-functionality/auth/ — Authentication and User Management

4.1 Login / Logout

  • [-] Login with valid credentials
  • [-] Login with invalid credentials — should display error message
  • [-] Logout — should redirect to login screen
  • [-] Auto-login enabled — should skip login screen
  • [-] Auto-login disabled — should display login screen
  • [-] Expired session — should redirect to login
  • [-] Session cleanup after logout

4.2 User Management (Admin)

  • [-] Admin creates new user
  • [-] Admin deactivates user
  • [-] Admin activates inactive user
  • [-] Admin renames user
  • [-] Admin changes user password
  • [-] Admin changes password — old password does not work after change
  • [-] Isolation flow: user A cannot see user B's flows

4.3 Global Variables (API Keys)

  • [-] Create global variable
  • Use global variable in component (API key)
  • [-] Edit existing global variable
  • [-] Delete global variable
  • [-] Create global variable of type "Generic"

core-functionality/knowledge-ingestion-management/ — Upload, Processing and Vectors

5.1 File Upload

  • [-] Upload file via component
  • [-] Upload files of different types (txt, pdf, json, py, wav)
  • [-] File size limit
  • [-] File management page

5.2 Processing and Vectorization

  • Document ingestion via Split Text + Embeddings component
  • Indexing in Vector Store — document available for query
  • Vector Store query returns relevant chunks for the prompt
  • Complete RAG pipeline (ingest → embed → store → retrieve → answer)

core-functionality/llm-agents/ — Agents and LLM Execution

⚠️ Tests in this section use SimpleAgentTemplatePage and are parameterized by model via models.json. Run npx playwright test tests/collect-models.spec.ts before executing these tests. See CLAUDE.md in this folder for the complete guide.

6.1 agent-component-regression.spec.ts — Agent Behavior Regression @stable

  • Agent responds without connected tools
  • Agent displays valid response and optionally reasoning steps
  • Stop button interrupts agent execution
  • Execution duration displayed after successful run
  • Response displayed progressively in the Playground (streaming)
  • Duration indicator displayed on canvas (node_duration_agent) after closing the playground
  • Agent responds to multiple consecutive messages in the same session

6.2 Other execution tests

  • [-] Agent displays reasoning steps in Playground → agent-reasoning-steps.spec.ts
  • [-] Composio (tool integration for Agent) → composio.spec.ts
  • Playground shows error when LLM run endpoint returns 500 (mocked invalid API key) → llm-agents/llm-invalid-api-key-ui.spec.ts
  • Playground input remains usable after API error (mocked) → llm-agents/llm-invalid-api-key-ui.spec.ts
  • Agent stops when configured stop condition is reached
  • Agent stops when maximum number of iterations is reached → agent-max-iterations.spec.ts
  • Agent with multiple configured tools executes correctly → agent-multi-tool-selection.spec.ts
  • Agent with configured timeout respects the limit
  • Switch provider in Agent → previous provider fields do not persist → agent-provider-field-isolation.spec.ts
  • Flow with Agent saved and reopened → settings preserved → agent-config-persistence.spec.ts
  • max_tokens truncates response as configured → agent-max-tokens.spec.ts
  • reasoning_effort field appears/disappears based on selected model → agent-reasoning-effort.spec.ts

6.3 Memory and Context

  • Memory Chatbot template loads with correct node and edge structure → llm-agents/memory-history-regression.spec.ts
  • Message History retains context between messages in the same Playground session → llm-agents/memory-history-regression.spec.ts
  • Session isolation: distinct session IDs have independent histories → llm-agents/memory-history-regression.spec.ts
  • Without Message History, LLM does not retain context between messages → llm-agents/memory-history-regression.spec.ts
  • n_messages parameter limits the number of retained messages → agent-n-messages-limit.spec.ts (confirmed bug: value saved correctly by frontend but ignored in backend execution)
  • Agent uses custom context_id — continuity between session messages → agent-context-id-continuity.spec.ts
  • Switching context_id isolates history between distinct sessions → agent-context-id-isolation.spec.ts

6.4 Tools and Integrations

  • Agent with integrated external MCP tool executes action and returns result
  • Agent executes multiple tools in sequence
  • Tool returns error — agent handles it and continues execution → agent-tool-error-handling.spec.ts
  • Multiple connected tools — agent selects the correct one for each prompt → agent-multi-tool-selection.spec.ts
  • Tool with invalid name — validation prevents execution with clear message → agent-tool-name-validation.spec.ts

6.5 Output and Reasoning

  • Inspect tools used by Agent in Playground
  • Agent returns output in structured JSON format (output_schema) → agent-structured-output.spec.ts
  • Agent returns output in correctly rendered Markdown
  • Agent Instructions (system prompt) is respected in the model response → agent-system-prompt.spec.ts
  • Input via direct field vs handle (ChatInput) — both work → agent-input-sources.spec.ts
  • Empty response or model refusal — component does not crash → agent-empty-refusal-response.spec.ts
  • Toggle add_current_date_tool works (enables/disables date tool) → agent-current-date-tool.spec.ts
  • handle_parsing_errors=False fails explicitly vs True auto-corrects → agent-parse-error-behavior.spec.ts
  • Image passed via input handle is processed correctly → agent-multimodal-image-input.spec.ts

core-functionality/model-provider/ — Provider Management

⚠️ Provider configuration tests via Settings use SettingsPage. See helpers/provider-setup/ for the setup helpers of each provider.

7.1 Provider Collection and Validation

  • [-] Validate API keys of all providers via real call → collect-models.spec.ts
  • [-] Collect available models per provider via UI → collect-models.spec.ts
  • Inactive providers appear as skipped in tests with reason → agent-component-regression.spec.ts
  • [-] Configure provider API key via Save Configuration (first setup) → collect-models.spec.ts
  • [-] Replace provider API key via Replace Configuration (existing key) → collect-models.spec.ts

7.2 OpenAI

  • [-] Configure OpenAI API key via GlobalVariables
  • [-] Select GPT model in agent
  • [-] Execute flow with OpenAI
  • Invalid API key error — display error message → core-functionality/llm-agents/provider-invalid-auth-error.spec.ts

7.3 Anthropic

  • [-] Configure Anthropic API key
  • [-] Select Claude model in agent
  • [-] Switch between Claude models (Sonnet, Haiku, Opus)
  • Invalid Anthropic API key error → core-functionality/llm-agents/provider-invalid-auth-error.spec.ts

7.4 Google Generative AI

  • [-] Configure Google API key in agent
  • [-] Select Gemini model in agent
  • Invalid Google API key error → core-functionality/llm-agents/provider-invalid-auth-error.spec.ts

7.5 Provider Management

  • [-] "Manage Model Providers" modal
  • [-] Available provider count
  • [-] Language Model component — configuration
  • [-] Model Input component
  • [-] Add new provider via modal
  • [-] Remove API key from existing provider

7.6 Open-Source Providers

  • Configure and execute flow with Ollama (local model)
  • Configure and execute flow with Groq
  • Configure and execute flow with Mistral

7.7 Model Parameters (Agent)

  • Temperature parameter (verify via network payload) → agent-max-tokens.spec.ts
  • Reasoning effort parameter — conditional field based on model → agent-reasoning-effort.spec.ts
  • Maximum token count — response truncated as configured → agent-max-tokens.spec.ts
  • Maximum agent iterations → agent-max-iterations.spec.ts
  • Use of custom context_id for memory isolation → agent-context-id-isolation.spec.ts
  • Output formatting (JSON via output_schema, Markdown, plain text) → agent-structured-output.spec.ts

core-functionality/observability-monitoring/ — Tracing, Logs and Metrics

8.1 Traces

  • [-] View execution traces
  • [-] Trace API returns paginated transactions
  • [-] Trace displays latency of each component
  • [-] Trace displays tokens consumed

8.2 Notifications

  • [-] System notifications
  • [-] Execution error notification
  • [-] Outdated component notification

8.3 User State

  • [-] Track user progress
  • [-] User flow state cleanup

8.4 Error Handling and Edge Cases

  • [-] Component that raises Python error
  • Flow with error displays appropriate message
  • [-] Network error during execution
  • [-] Execution timeout — clear message to user

core-functionality/playground/ — Chat, Rendering and Output Tests

9.1 Chat Interactions

  • Open Playground → exercised by every @stable playground spec via playground-btn-flow-io
  • Send text message → exercised by playground-ux.spec.ts, playground-message-edit.spec.ts, playground-session-nav.spec.ts and others
  • Receive LLM response → exercised by all specs that send a message via ChatInput → ChatOutput echo flow
  • [-] Response streaming (SSE) → no dedicated spec; exercised implicitly by playground-ux.spec.ts
  • [-] Response polling → no dedicated spec
  • [-] Direct response → no dedicated spec
  • Playground UX (playground-ux) → playground/playground-ux.spec.ts
  • Send empty message — should disable send button → playground/playground-empty-message-send.spec.ts (BUG: button enabled even when empty)
  • Send message while response is in progress — should wait or queue
  • Attach image in chat — compact preview appears in input before sending → core-functionality/playground/playground-output-image.spec.ts
  • Image rendered in user message bubble after sending → core-functionality/playground/playground-output-image.spec.ts

9.2 History and Session

  • [-] Configure custom session ID → playground/playground-session-id.spec.ts (needs rewrite — see issue)
  • Switch session — messages are isolated per session → core-functionality/playground/playground-session-nav.spec.ts
  • Edit user message — hover reveals edit button, saved changes replace original text → core-functionality/playground/playground-message-edit.spec.ts
  • Cancel message edit — original text is preserved → core-functionality/playground/playground-message-edit.spec.ts
  • Message edited in playground is reflected in Session Logs → core-functionality/playground/playground-message-edit.spec.ts
  • Clear chat removes all messages from Default Session (clear-chat-option via header menu) → core-functionality/playground/playground-session-clear.spec.ts
  • Clear full session history (Default session) → playground/playground-clear-history.spec.ts
  • Delete user-created session → playground/playground-clear-history.spec.ts
  • History persists when reopening Playground → llm-agents/memory-history-regression.spec.ts, core-functionality/playground/playground-history-persist.spec.ts
  • Rename unavailable for the Default Session → core-functionality/playground/playground-session-rename.spec.ts
  • Rename unavailable for a session with no messages → core-functionality/playground/playground-session-rename.spec.ts
  • Rename available and functional for a session with messages (Enter confirms, Escape cancels) → core-functionality/playground/playground-session-rename.spec.ts
  • Create new session via new-chat button → core-functionality/playground/playground-session-nav.spec.ts
  • Switch between sessions via session selector sidebar → core-functionality/playground/playground-session-nav.spec.ts
  • Open Message Logs via session more-menu → core-functionality/playground/playground-message-logs.spec.ts
  • Delete messages inside Message Logs table → core-functionality/playground/playground-message-logs.spec.ts
  • Select individual session checkbox → reveals bulk-delete-button → core-functionality/playground/playground-bulk-delete.spec.ts
  • Select all non-default sessions via select-all-checkbox → core-functionality/playground/playground-bulk-delete.spec.ts
  • Bulk delete selected sessions → Default Session preserved → core-functionality/playground/playground-bulk-delete.spec.ts

9.3 Advanced Playground Features

  • Playground fullscreen mode → playground/playground-fullscreen.spec.ts
  • [-] Shareable Playground — URL generation validated (switch enables sharing, href matches /playground/uuid) → playground/playground-shareable-url.spec.ts
  • [!] Voice mode (voice assistant) → ui-ux/voice-assistant.spec.ts (all tests unconditionally skipped — spec is a stub)
  • Stop button in Playground → core-functionality/playground/stop-button-playground.spec.ts

9.4 Output Modal

  • [-] Copy component output → playground/output-modal-copy-button.spec.ts (needs rewrite — see issue)
  • [-] Copy button in output → playground/output-modal-copy-button.spec.ts (needs rewrite — see issue)

9.5 Structured Data Output

  • JSON Data output renders as code block → core-functionality/playground/playground-output-data.spec.ts
  • DataFrame output renders as Markdown table → core-functionality/playground/playground-output-data.spec.ts

core-functionality/project-management/ — Project and Folder Management

10.1 Folder CRUD

  • [-] Create new folder
  • [-] Rename folder
  • [-] Delete empty folder
  • [-] Delete folder with flows inside
  • [-] Integrity after deletion
  • [-] Create folder after deleting all folders
  • [-] Upload flow by drag-and-drop to folder
  • [-] Move flow to another folder

10.2 Folder Navigation

  • [~] Navigate between folders
  • [-] Search flow by name filters results correctly
  • [-] Folders in navigation sidebar

core-functionality/templates/ — Predefined Flow and Component Models

11.1 Basic Templates

  • [-] Basic Prompting (OpenAI)
  • [-] Basic Prompting (Anthropic)
  • [-] Simple Agent (OpenAI)
  • [-] Simple Agent (Anthropic)
  • [-] Simple Agent with memory
  • [-] Vector Store RAG
  • Memory Chatbot
  • [-] Basic Prompting (OpenAI) → core/integrations/Basic Prompting.spec.ts
  • [-] Basic Prompting (Anthropic) → core/integrations/Basic Prompting Anthropic.spec.ts
  • [-] Simple Agent (OpenAI) → core/integrations/Simple Agent.spec.ts
  • [-] Simple Agent (Anthropic) → core/integrations/Simple Agent Anthropic.spec.ts
  • [-] Simple Agent with memory → core/integrations/Simple Agent Memory.spec.ts
  • [-] Vector Store RAG → core/integrations/Vector Store.spec.ts
  • Memory Chatbot → llm-agents/memory-history-regression.spec.ts

11.2 Content Generation Templates

  • [-] Blog Writer
  • [-] Instagram Copywriter
  • [-] Twitter Thread Generator
  • [-] SEO Keyword Generator
  • [-] Portfolio Website Code Generator
  • [-] SaaS Pricing

11.3 Analysis and Processing Templates

  • [-] Document QA
  • [-] Invoice Summarizer
  • [-] Financial Report Parser
  • [-] Image Sentiment Analysis
  • [-] Text Sentiment Analysis
  • [-] Youtube Analysis

11.4 Agent Templates

  • [-] Dynamic Agent
  • [-] Hierarchical Agent
  • [-] Sequential Task Agent
  • [-] Social Media Agent
  • [-] Travel Planning Agent
  • [-] Market Research
  • [-] Research Translation Loop
  • [-] Pokedex Agent
  • [-] Price Deal Finder
  • [-] News Aggregator

11.5 Advanced Templates

  • [-] Custom Component Generator
  • [-] Prompt Chaining
  • [-] Decision Flow
  • [-] Similarity
  • [-] MCP Server (starter projects)

flow-functionality/ — Graph Execution, Drag-and-Drop and JSON

12.1 Create Flow

  • [-] Create blank flow
  • [-] Create flow from template
  • [-] Create flow by duplicating an existing one
  • [-] Create flow via JSON file import

12.2 View and Edit Flow

  • [-] Rename flow via editor header
  • [-] Edit flow name and description
  • [-] Flow auto-save on changes
  • [-] Flow settings

12.3 Delete Flow

  • [-] Delete individual flow
  • [-] Delete multiple flows (bulk actions)
  • [-] Confirm deleted flow does not appear in listing

12.4 Export / Import Flow

  • [-] Export flow as JSON
  • [-] Import flow via JSON file upload
  • [~] Import flow with outdated components
  • [-] Import invalid JSON — should display error message

12.5 Flow Operations

  • [-] Lock flow — prevents editing
  • [-] Unlock flow
  • [-] Move flow between folders via API
  • [-] Publish flow → flow-functionality/publish-flow.spec.ts
  • [-] Save flow components as template

12.6 Flow Execution

  • [-] Execute flow via Run button → core/features/run-flow.spec.ts
  • [-] Stop building flow → core/features/stop-building.spec.ts
  • [!] Playground button disabled with empty flow — needs review → regression/flow-functionality/generalBugs-shard-3.spec.ts (test skipped: assertion was a no-op, current Langflow behavior to confirm)

mcp/ — Model Context Protocol

⚠️ Tests that execute agents via MCP must use SimpleAgentTemplatePage and models.json. See CLAUDE.md in this folder for the complete guide.

mcp/client/ — Tool and Context Consumption

13.1 MCP Client

  • Configure connection with external MCP server (stdio or HTTP)
  • List available tools via MCP protocol
  • Execute MCP server tool and receive result in flow
  • List available resources via MCP protocol
  • Consume resource URI and inject content into flow
  • MCP server connection error displays clear message

mcp/server/ — Resource and Tool Provider

14.1 MCP Server

  • [-] MCP Server tab in flow
  • [-] Add MCP server via modal
  • [-] Starter project with MCP
  • Flow exposed as MCP server — verify generated endpoint
  • Execute MCP server tool via MCP protocol
  • Resource exposed by server is accessible via URI
  • Prompt exposed by server returns correct template

ui-ux/ — Visual Interface, Canvas and Design System

15.1 Component Sidebar

  • [-] Search component by name
  • [-] Hover over component shows tooltip/preview
  • [-] Keyboard search (keyboard shortcut)
  • [-] Filter components by category
  • [-] Sidebar shows correct provider count

15.2 Add Components to Canvas

  • [-] Drag component from sidebar to canvas
  • [-] Double-click in sidebar adds component to canvas
  • [-] Hover + click "+" button adds component to canvas
  • [-] Added component appears with default settings

15.3 Component Connections

  • [-] Connect two compatible components
  • [-] Prevent connection between incompatible types
  • [-] Delete edge/connection
  • [-] Filter edges by data type
  • [-] Reconnect existing edge

15.4 Node Manipulation

  • [-] Delete component from canvas
  • [-] Copy and paste component (Ctrl+C / Ctrl+V)
  • [-] Canvas keyboard shortcuts
  • [-] Minimize component on canvas
  • [-] Move component within canvas
  • [-] Select multiple components via box selection
  • [-] Delete multiple selected components
  • [-] Deselect node by clicking on empty canvas area
  • [-] Deselect node via Escape

15.5 Canvas Zoom and Navigation

  • [-] Zoom in / Zoom out
  • [-] Fit View centers nodes
  • [-] Fit View button in toolbar
  • [-] Scroll to navigate canvas
  • [~] Minimap — feature flag-gated

15.6 Grouping

  • [-] Create component group
  • [-] Ungroup components
  • [-] Expand/collapse group

15.7 Freeze and State

  • [-] Freeze component
  • [-] Freeze path
  • [-] Unfreeze component

15.8 Sticky Notes

  • [-] Add sticky note
  • Edit sticky note text
  • [-] Change sticky note color
  • [-] Resize sticky note
  • [-] Delete sticky note

15.9 Right-Click and Menus

  • [-] Context menu via right-click on canvas
  • [-] Context menu via right-click on component
  • [-] Main menu actions

15.10 Settings and UI Configuration

  • [-] Access Settings page
  • [-] Message history settings
  • Change appearance/theme settings — dark/light toggle updates #body.dark class → ui-ux/settings-theme-toggle.spec.ts
  • [-] Keyboard shortcuts work in editor
  • [~] All documented shortcuts work

Coverage Summary — Test Automation Coverage

Validated = test carries the @stable tag. Needs validation = automated but not yet @stable (bug, flake under investigation, or pending team review).

Module Total Validated [x] Needs validation [-] Partial [~]/[!] Not automated [ ]
api/flows/ — REST API 21 0 21 0 0
core-components/ — Component Config 22 0 20 0 2
core-components/ — Core Components 43 28 10 0 5
core-functionality/auth/ 19 0 18 0 1
core-functionality/knowledge-ingestion/ 8 0 4 0 4
core-functionality/llm-agents/ 40 13 2 0 25
core-functionality/model-provider/ 31 4 18 0 9
core-functionality/observability-monitoring/ 13 0 12 0 1
core-functionality/playground/ 38 29 7 1 1
core-functionality/project-management/ 11 0 10 1 0
core-functionality/templates/ 41 2 39 0 0
flow-functionality/ 23 0 21 2 0
mcp/client/ 6 0 0 0 6
mcp/server/ 7 0 3 0 4
ui-ux/ — Canvas 42 0 40 1 1
ui-ux/ — Settings 5 1 3 1 0
TOTAL 370 77 (21%) 228 (62%) 6 (2%) 59 (16%)

Note: Validated [x] = checklist bullets, not unique tests. A single @stable test may cover multiple bullets (e.g. the agent suite covers 7 bullets via test.step()). The canonical list of 51 unique @stable tests is in Phase 0 — Validated below.

Implementation Roadmap

🟢 Phase 0 — Validated

Tests carrying the @stable tag — included in the weekly stable workflow. 51 tests across 20 spec files.

core-components/

  • Loop component — renders correctly with all handles and output inspection buttons → core-components/loop-component-regression.spec.ts
  • Loop component — run without connections shows build failed notification → core-components/loop-component-regression.spec.ts
  • Loop component — Research Translation Loop template: full wiring and iterates over 2 ArXiv papers → core-components/loop-component-regression.spec.ts
  • Webhook component — HTTP POST accepts JSON and plain-text bodies returning 202 → core-components/webhook-component-regression.spec.ts
  • Webhook component — flow is saved to database and contains the Webhook node → core-components/webhook-component-regression.spec.ts
  • Webhook component — cURL command in inspector shows valid POST URL with flow ID → core-components/webhook-component-regression.spec.ts
  • Webhook component — empty data field returns empty Data object → core-components/webhook-component-regression.spec.ts
  • Webhook component — endpoint field renders the actual webhook URL → core-components/webhook-component-regression.spec.ts
  • Webhook component — copy button copies the endpoint URL to clipboard → core-components/webhook-component-regression.spec.ts
  • Webhook component — POST to non-existent flow name returns 404 → core-components/webhook-component-regression.spec.ts
  • Webhook component — valid JSON payload is propagated as structured Data output → core-components/webhook-component-regression.spec.ts
  • Webhook component — invalid JSON payload is encapsulated in {payload: ...} → core-components/webhook-component-regression.spec.ts
  • GET /api/v1/monitor/messages returns 200 with array response → core-components/webhook-component-regression.spec.ts

core-functionality/llm-agents/

  • agent interaction suite (parameterized per provider/model: tools-free response, reasoning steps, streaming + duration, multiple consecutive messages, canvas duration indicator) → agent-component-regression.spec.ts
  • agent stop button must halt execution mid-run → agent-component-regression.spec.ts
  • playground shows error when LLM run endpoint returns 500 (mocked invalid API key) → llm-invalid-api-key-ui.spec.ts
  • playground input remains usable after API error (mocked) → llm-invalid-api-key-ui.spec.ts
  • memory chatbot template loads with correct node structure → memory-history-regression.spec.ts
  • message history context retention suite (within-session retention, accumulation, persistence after reopen) → memory-history-regression.spec.ts
  • session isolation: new session has no context from previous session → memory-history-regression.spec.ts
  • should display error message when using invalid authentication for provider (parameterized per provider) → provider-invalid-auth-error.spec.ts

core-functionality/playground/

  • selecting an individual session checkbox must reveal the bulk-delete-button → playground-bulk-delete.spec.ts
  • select-all-checkbox must select all non-default sessions → playground-bulk-delete.spec.ts
  • bulk-delete-button must remove all selected sessions from the sidebar → playground-bulk-delete.spec.ts
  • clear chat on Default session must remove messages but keep the session → playground-clear-history.spec.ts
  • deleting a user-created session must remove it and return to Default session → playground-clear-history.spec.ts
  • send button is enabled when input is empty (Langflow bug) → playground-empty-message-send.spec.ts
  • send button becomes enabled after typing a message → playground-empty-message-send.spec.ts
  • clearing the input after typing leaves the field empty → playground-empty-message-send.spec.ts
  • playground opens in fullscreen with chat input visible → playground-fullscreen.spec.ts
  • playground closes and reopens correctly from the flow editor → playground-fullscreen.spec.ts
  • edit user message — hover reveals edit button and saved changes replace original text → playground-message-edit.spec.ts
  • cancel message edit — original text is preserved → playground-message-edit.spec.ts
  • message edited in playground is reflected in Session Logs → playground-message-edit.spec.ts
  • message-logs-option must open the Session Logs modal for the active session → playground-message-logs.spec.ts
  • selecting messages in the log table and deleting them must reduce the row count → playground-message-logs.spec.ts
  • playground must render JSON Data output as a code block → playground-output-data.spec.ts
  • playground must render DataFrame output as a markdown table → playground-output-data.spec.ts
  • playground must show image compact preview in input area after attaching an image → playground-output-image.spec.ts
  • playground must display uploaded image in user message after sending → playground-output-image.spec.ts
  • clear-chat removes all messages from Default Session → playground-session-clear.spec.ts
  • new-chat button must add a new session entry to the sidebar → playground-session-nav.spec.ts
  • session selector sidebar must switch to the selected session → playground-session-nav.spec.ts
  • rename option must not be available for the Default Session → playground-session-rename.spec.ts
  • rename option must not be available for a session with no messages → playground-session-rename.spec.ts
  • rename option must be available and functional for a session with messages → playground-session-rename.spec.ts
  • user message must appear instantly in playground before AI responds → playground-ux.spec.ts
  • playground must scroll to latest message after sending → playground-ux.spec.ts
  • playground input field must be ready after flow responds → playground-ux.spec.ts
  • User must be able to stop building from inside Playground → stop-button-playground.spec.ts

ui-ux/

  • dark and light mode toggle correctly updates the body class → settings-theme-toggle.spec.ts

🔵 Phase 1 — Next Delivery

Validate ([-]) and create ([ ]) in the modules below. See details in Part II.

Module Validate ([-]) Create ([ ])
api/flows/ — REST API 21 0
core-components/ — Component Config 20 2
core-components/ — Core Components 18 9
core-functionality/auth/ 18 1
core-functionality/llm-agents/ 2 25
core-functionality/model-provider/ 18 9
core-functionality/playground/ 7 1
mcp/client/ 0 6
mcp/server/ 3 4
ui-ux/ — Canvas 40 1

🟡 Phase 2 — Next Delivery

Remaining modules after Phase 1 completion. See details in Part II.

Module Validate ([-]) Create ([ ])
core-functionality/observability-monitoring/ 12 1
core-functionality/knowledge-ingestion/ 4 4
flow-functionality/ 20 0
core-functionality/project-management/ 10 0
core-functionality/templates/ 39 0
ui-ux/ — Settings 3 0