Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 21 additions & 7 deletions QA-CHECKLIST.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,15 +165,23 @@
- [x] Dismiss button on the Files field clears the value → `core-components/chat-input-files-field-regression.spec.ts`

#### 3.2 Prompt Template
- [x] Prompt with variables in curly braces → `core-components/prompt-template-component-regression.spec.ts`
- [x] Prompt modal opens, accepts edits, and changes persist → `core-components/prompt-template-component-regression.spec.ts`
- [x] Dynamic port generated when adding a variable to the prompt → `core-components/prompt-template-component-regression.spec.ts`
- [x] Removing a variable from the prompt deletes the corresponding port → `core-components/prompt-template-component-regression.spec.ts`
- [x] Prompt Template renders on canvas with output handle → `core-components/prompt-template-component-regression.spec.ts`
- [x] Variables in curly braces generate dynamic input handles → `core-components/prompt-template-component-regression.spec.ts`
- [x] Removing a variable removes its input handle → `core-components/prompt-template-component-regression.spec.ts`
- [x] Replacing a variable updates handles accordingly → `core-components/prompt-template-component-regression.spec.ts`
- [x] Clearing the template removes all dynamic handles → `core-components/prompt-template-component-regression.spec.ts`
- [x] Modal edits persist in UI and in saved flow → `core-components/prompt-template-component-regression.spec.ts`
- [x] `use_double_brackets` toggle is exposed in the InspectionPanel with its upstream display name → `core-components/prompt-template-double-brackets-regression.spec.ts`
- [x] Default toggle state is OFF; f-string mode extracts `{var}` and treats `{{var}}` as literal → `core-components/prompt-template-double-brackets-regression.spec.ts`
- [x] Enabling toggle switches parser to mustache mode; `{{var}}` creates handle and `{var}` is ignored → `core-components/prompt-template-double-brackets-regression.spec.ts`
- [x] Disabling toggle reverts to f-string mode and variables are re-extracted under the new parser → `core-components/prompt-template-double-brackets-regression.spec.ts`
- [x] `use_double_brackets` value persists in the autosaved flow → `core-components/prompt-template-double-brackets-regression.spec.ts`
- [x] f-string parser rejects `{var.attr}` (dot notation) with an error toast and creates no handle → `core-components/prompt-template-invalid-patterns-regression.spec.ts`
- [x] f-string parser rejects `{var name}` (space inside identifier) with an error toast and creates no handle → `core-components/prompt-template-invalid-patterns-regression.spec.ts`
- [x] f-string parser rejects `{var,name}` (comma inside identifier) with an error toast and creates no handle → `core-components/prompt-template-invalid-patterns-regression.spec.ts`
- [x] f-string parser rejects `{1var}` (leading digit) with an error toast and creates no handle → `core-components/prompt-template-invalid-patterns-regression.spec.ts`
- [x] f-string parser accepts `{}` (empty braces) silently — no error, no handle → `core-components/prompt-template-invalid-patterns-regression.spec.ts`
- [x] f-string parser deduplicates repeated variables — `{name} and {name}` yields exactly one handle → `core-components/prompt-template-invalid-patterns-regression.spec.ts`
Comment on lines +179 to +184

#### 3.3 API Request (HTTP)
- [x] Renders on canvas with URL and API Response handles → `core-components/api-request-component-regression.spec.ts`
Expand Down Expand Up @@ -691,7 +699,7 @@
|--------|-------|-----------------|------------------------|---------------------|---------------------|
| `api/flows/` — REST API | 21 | 2 | 19 | 0 | 0 |
| `core-components/` — Component Config | 22 | 1 | 19 | 0 | 2 |
| `core-components/` — Core Components | 55 | 47 | 3 | 1 | 4 |
| `core-components/` — Core Components | 63 | 55 | 3 | 1 | 4 |
| `core-functionality/auth/` | 19 | 0 | 18 | 0 | 1 |
| `core-functionality/knowledge-ingestion/` | 8 | 0 | 4 | 0 | 4 |
| `core-functionality/llm-agents/` | 40 | 13 | 2 | 0 | 25 |
Expand All @@ -705,7 +713,7 @@
| `mcp/server/` | 7 | 0 | 3 | 0 | 4 |
| `ui-ux/` — Canvas | 43 | 2 | 39 | 1 | 1 |
| `ui-ux/` — Settings | 5 | 1 | 3 | 1 | 0 |
| **TOTAL** | **393** | **115 (29%)** | **213 (54%)** | **7 (2%)** | **58 (15%)** |
| **TOTAL** | **401** | **123 (31%)** | **213 (53%)** | **7 (2%)** | **58 (14%)** |

> Note: `Validated [x]` counts checklist bullets, not `test()` calls. The
> `@stable` tag is per-`test()`, and a single `@stable` test may map to
Expand All @@ -721,7 +729,7 @@

### 🟢 Phase 0 — Validated

> 110 `test()` calls carrying the `@stable` tag, distributed across 36 spec
> 116 `test()` calls carrying the `@stable` tag, distributed across 37 spec
> files. Run weekly by the stable workflow. New specs are merged with all
> tests tagged `@stable`; the tag is removed per-test during weekly triage
> when a failure is classified as a test bug — so a spec may end up with a
Expand Down Expand Up @@ -778,6 +786,12 @@
- [x] Prompt Template — enabling toggle switches parser to mustache mode; {{var}} creates handle and {var} is ignored → `prompt-template-double-brackets-regression.spec.ts`
- [x] Prompt Template — disabling toggle reverts to f-string mode and variables are re-extracted under the new parser → `prompt-template-double-brackets-regression.spec.ts`
- [x] Prompt Template — use_double_brackets value persists in the autosaved flow → `prompt-template-double-brackets-regression.spec.ts`
- [x] Prompt Template — `{var.attr}` (dot notation) is rejected with an error toast and creates no handle → `prompt-template-invalid-patterns-regression.spec.ts`
- [x] Prompt Template — `{var name}` (space inside identifier) is rejected with an error toast and creates no handle → `prompt-template-invalid-patterns-regression.spec.ts`
- [x] Prompt Template — `{var,name}` (comma inside identifier) is rejected with an error toast and creates no handle → `prompt-template-invalid-patterns-regression.spec.ts`
- [x] Prompt Template — `{1var}` (leading digit) is rejected with an error toast and creates no handle → `prompt-template-invalid-patterns-regression.spec.ts`
- [x] Prompt Template — `{}` (empty braces) is accepted by the parser and creates no handle → `prompt-template-invalid-patterns-regression.spec.ts`
- [x] Prompt Template — repeating the same variable produces exactly one handle (deduplication contract) → `prompt-template-invalid-patterns-regression.spec.ts`
- [x] User should be able to use components as tool → `tool-mode.spec.ts`
- [x] Webhook component — cURL command in inspector shows valid POST URL with flow ID → `webhook-component-regression.spec.ts`
- [x] Webhook component — empty data field returns empty Data object → `webhook-component-regression.spec.ts`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ Tests 2–6 use the helper `setPromptTemplate(page, value)` which:
- Prompt rendering/execution behind an LLM (covered by `llm-agents` specs such as `memory-history-regression.spec.ts`)
- Tool Mode interaction (covered by `tool-mode.spec.ts`)
- Cross-component data flow (covered by `flow-functionality/` specs)
- Variable name validation (e.g., reserved keywords, special characters)
- Invalid-character rejection in variable names (e.g. `{var.attr}`, `{var name}`, `{1var}`), the empty-braces contract, and variable deduplication (covered by `prompt-template-invalid-patterns-regression.spec.ts`)
- The `use_double_brackets` toggle and the mustache-mode parser (covered by `prompt-template-double-brackets-regression.spec.ts`)

---
Expand Down
132 changes: 132 additions & 0 deletions docs/core-components/prompt-template-invalid-patterns-regression.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
# Prompt Template — Invalid Patterns Regression

**Last validated:** Langflow 1.10.x

---

## What this test validates *(required)*

Validates the **f-string parser contract** of the Prompt Template component when the template string contains patterns that the upstream validator either rejects or treats with specific positive-path semantics. The companion happy-path coverage lives in `prompt-template-component-regression.spec.ts`; this spec focuses on the negative-path and the dedup contract.

The suite has 6 tests covering 4 rejection paths and 2 positive-path defensive assertions:

**Rejection (toast surfaces, no handle created)**

1. **`{var.attr}` (dot notation)** — the dot character is in the upstream `_INVALID_CHARACTERS` set. `_check_input_variables` raises `ValueError("Input variables contain invalid characters or formats. ...")`.
2. **`{var name}` (space inside identifier)** — same path as #1; space is in `_INVALID_CHARACTERS`.
3. **`{var,name}` (comma inside identifier)** — same path; comma is in `_INVALID_CHARACTERS`. Catches the common templating mistake where users write `{a,b}` thinking it declares multiple variables.
4. **`{1var}` (leading digit)** — the upstream `_fix_variable` helper specifically flags identifiers starting with a digit. Anchored on the toast fragment `"Invalid variables: 1"` to confirm the leading-digit branch fired (not just an incidental match on the digit character).

**Positive-path defensive**

5. **`{}` (empty braces)** — Python's `Formatter().parse()` yields `("", "", "", None)` and `extract_input_variables_from_prompt` filters empty `field_name` out. The save succeeds, no error toast, no dynamic handle. This test exists so that if a future change starts rejecting `{}` or extracting it as a real variable, the regression surfaces immediately.
6. **`{name} and {name}` (duplicate)** — `extract_input_variables_from_prompt` tracks seen field names in a `set`, so a repeated `{name}` yields exactly one input variable. Asserts exactly one `name` handle is rendered, not two.

> **Note on scope.** The originally proposed cases `{}` rejection and `{var-name}` rejection (from issue #214) were dropped after probing the live `/api/v1/validate/prompt` endpoint: neither pattern raises today — `{}` is filtered out by the formatter and the hyphen character is **not** in `_INVALID_CHARACTERS`. The replaced rejection cases (`{var name}`, `{var,name}`, `{1var}`) were chosen because they exercise distinct branches of `_check_input_variables`: the `_INVALID_CHARACTERS` set check (space, comma) and the `_fix_variable` leading-digit branch.

If any of these tests fails, one of three contracts has regressed: the rejection of invalid characters in variable names, the dedup behavior of the parser, or the empty-braces / format-positional escape handling.

---

## Tags *(required)*

All 6 tests: `@stable` `@regression` `@components`

None carry `@release` — these are defensive contract assertions, not happy-path flows.

---

## Step by step *(required)*

Every test starts with the same `addPromptComponent(page)` helper used in `prompt-template-component-regression.spec.ts` (blank flow → sidebar search → add → adjust view → assert 1 node).

### Rejection tests (tests 1–4)

The four rejection cases all run the same step sequence via the `runRejectionContract` helper:

1. `addPromptComponent(page)`.
2. Open the prompt modal, fill the textarea with the invalid template, click `genericModalBtnSave` (helper `fillAndSavePromptTemplate`). The helper does **not** wait for modal close — the modal stays open on error.
3. Assert the error toast (`.error-build-message`) is visible within 5 s (the toast auto-dismisses after 5 s — see `src/frontend/src/alerts/error/index.tsx:22`).
4. Assert the toast text contains:
- The constant title `"There is something wrong with this prompt"` (from i18n `errors.prompt`).
- The upstream-error fragment `"Input variables contain invalid characters or formats"`.
- A per-case fragment that anchors the test to the offending pattern — `var.attr`, `var name`, `var,name`, or `"Invalid variables: 1"` for the leading-digit case.
5. Assert the textarea (`modal-promptarea_prompt_template`) is still visible — the frontend sets `isEdit=true` in the `onError` callback so the user can correct without losing input.
6. Press `Escape` to leave the test in a clean state.
7. (Final body-level check) Assert `dynamicHandlesLocator` count is exactly 0 — the rejected save never reached the node.

Note on the fixture: the save deliberately returns HTTP 500, but `tests/fixtures/fixtures.ts` only fails on `flow_error`-type events from `/build/`, `/run/`, or `/events?event_delivery=`. HTTP 500s on `/api/v1/validate/prompt` are logged as `http_error` and do not fail the test, so no `page.allowFlowErrors()` opt-out is needed.

### 5. `{}` is accepted by the parser and creates no handle
- `addPromptComponent(page)`.
- `fillAndSavePromptTemplate(page, "Plain {} text")`.
- Assert the textarea is hidden within 10 s (save succeeded — the modal closed).
- Assert error toast count is 0.
- Assert dynamic-handle count is 0.

### 6. Deduplication: `{name} and {name}` produces one handle
- `addPromptComponent(page)`.
- `fillAndSavePromptTemplate(page, "Hello {name}, goodbye {name}.")`.
- Assert the textarea is hidden within 10 s.
- Assert `handle-prompt template-shownode-name-left` is visible.
- Assert dynamic-handle count is exactly 1.

---

## Validation criterion *(required)*

- For each invalid pattern that hits `_INVALID_CHARACTERS` (dot, space, comma) or `_fix_variable`'s leading-digit branch:
- The error toast `.error-build-message` is visible and contains the upstream `ValueError` title (`errors.prompt`) plus the message fragment `"Input variables contain invalid characters or formats"`.
- The toast also includes a per-case fragment anchored on the offending pattern (the variable identifier for set-check failures, or `"Invalid variables: 1"` for the leading-digit branch), confirming the right code path fired.
- The dynamic handle count on the canvas remains 0 — the rejected save did not propagate to the node.
- The modal stays open in edit mode (textarea visible) so the user can correct the input.
- For `{}` (empty braces): the save closes the modal, no error toast appears, and no handle is created.
- For `{name} and {name}`: the save closes the modal and exactly one `name` handle is rendered.

---

## External dependencies *(required)*

- `src/lfx/src/lfx/base/prompts/api_utils.py` — `_INVALID_CHARACTERS` set, `_fix_variable` (leading-digit branch), `_check_input_variables`, `_check_for_errors`, `validate_prompt`. Drives the entire rejection contract; tests 1–3 exercise the set-check branch (dot, space, comma), test 4 exercises the leading-digit branch. Any change to the set, the leading-digit logic, or the message format breaks tests 1–4.
- `src/lfx/src/lfx/interface/utils.py` — `extract_input_variables_from_prompt`: the dedup `set` and the empty `field_name` filter back tests 5 and 6.
- `src/backend/base/langflow/api/v1/validate.py` — `POST /validate/prompt`: returns HTTP 500 with `detail=str(ValueError(...))` on rejection. The HTTP status and the `detail` shape are what the frontend's `onError` callback consumes.
- `src/frontend/src/modals/promptModal/index.tsx` — `genericModalBtnSave`, `modal-promptarea_prompt_template`, and the `usePostValidatePrompt` mutation's `onError` callback that maps the API error into the toast (title from `t("errors.prompt")`, detail from `error.response.data.detail`). Also sets `isEdit=true` so the modal stays open on error.
- `src/frontend/src/alerts/error/index.tsx` — `ErrorAlert` component renders with CSS class `.error-build-message`; the 5-second auto-dismiss timeout there defines the maximum window in which the toast assertion must run.
- `src/frontend/src/locales/en.json` — i18n key `errors.prompt` whose value `"There is something wrong with this prompt, please review it"` is the source for the toast title. All four rejection tests (1–4) assert via `toContainText("There is something wrong with this prompt")` — a substring check, not a full-string equality. Localizing this prefix (or shortening it past the asserted substring) would break tests 1–4.

---

## What this test does not cover *(optional)*

- Variable extraction from valid templates and dynamic-handle rendering (covered by `prompt-template-component-regression.spec.ts`)
- The `use_double_brackets` toggle and the f-string escape behavior `{{var}}` rendering as a literal (covered by `prompt-template-double-brackets-regression.spec.ts`, where the two parser modes are asserted side by side and the comparison is informative)
- Invalid patterns in **mustache mode** (`{{ var }}`, `{{var.attr}}`, `{{#section}}{{/section}}`, `{{{var}}}`) — tracked as a follow-up issue, to be covered in a dedicated spec because the toggle flips a different code path (`validate_mustache_template`)
- LLM/runtime errors when a flow built on top of the Prompt Template is executed
- Tool Mode interaction (covered by `tool-mode.spec.ts`)

---

## Preconditions *(optional)*

- Langflow running and accessible at `PLAYWRIGHT_BASE_URL`
- No API key required — the Prompt Template component is a pure templating layer with no LLM calls
- The fixture from `tests/fixtures/fixtures.ts` is used. The HTTP 500 the rejection cases trigger on `/api/v1/validate/prompt` is logged as `http_error` but does not fail the test (the fixture only fails on `flow_error`-type events from `/build/`, `/run/`, `/events?event_delivery=`).

---

## When to review this test *(optional)*

- If `_INVALID_CHARACTERS` in `src/lfx/src/lfx/base/prompts/api_utils.py` changes (add/remove a character) — the rejection tests are coupled to that set.
- If the error message format in `_check_for_errors` changes — tests 1–4 assert the leading fragment `"Input variables contain invalid characters or formats"`, and test 4 additionally anchors on `"Invalid variables: 1"`.
- If the i18n key `errors.prompt` is renamed or its English value shortened past the asserted prefix `"There is something wrong with this prompt"` — tests 1–4 assert that substring.
- If the frontend stops calling `setIsEdit(true)` in `onError` — step 5 of the rejection contract asserts the textarea stays visible.
- If the toast component's CSS class changes from `.error-build-message` to something else, or a `data-testid` is added (the spec could then anchor on it instead of the class).

---

## Notes *(optional)*

- The validate endpoint deliberately returns HTTP 500 for the rejection cases, but the fixture only fails on `flow_error`-type events from `/build/`, `/run/`, or `/events?event_delivery=` (see `tests/fixtures/fixtures.ts:230–262`). The 500 from `/api/v1/validate/prompt` is logged as `http_error` and is harmless — no `page.allowFlowErrors()` opt-out is needed here. Other specs (`loop-component-regression.spec.ts`, `api-request-component-regression.spec.ts`) do call `allowFlowErrors()` because they exercise the `/build/` and `/run/` endpoints, which would actually fail the test.
- The error toast auto-dismisses after 5 seconds (`src/frontend/src/alerts/error/index.tsx:22`). The first toast assertion uses a 5-second timeout to match — running additional waits before the toast assertion would race against the dismissal.
- The originally proposed `{var-name}` rejection case was dropped because hyphen is not in `_INVALID_CHARACTERS` upstream — `{var-name}` actually creates a `var-name` handle. Probed against `POST /api/v1/validate/prompt` on Langflow 1.10.x. If Langflow ever adds hyphen to the invalid-chars set, this spec should grow a case for it.
- The originally proposed `{}` rejection case was reframed as a positive-path defensive test for the same reason: Python's `Formatter().parse()` yields an empty `field_name` that the extractor filters out, so the save succeeds silently. The test asserts that contract.
Loading
Loading