test(core-components): cover invalid f-string patterns in prompt template - #220
Merged
Victor-w-Madeira merged 5 commits intoMay 12, 2026
Merged
Conversation
…late
Adds prompt-template-invalid-patterns-regression.spec.ts with 5 tests
asserting the f-string parser contract for the Prompt Template
component: three rejection paths surfaced as error toasts
(`{var.attr}` dot notation, `{var name}` space inside identifier,
`{1var}` leading digit) plus two positive-path defensive cases
(`{}` empty braces filtered out by the formatter; `{name} and {name}`
deduplicated to one handle).
Issue #214 originally listed `{}` and `{var-name}` rejection — both
were dropped after probing `/api/v1/validate/prompt` directly:
hyphen is not in `_INVALID_CHARACTERS` and empty braces are filtered
by Python's `Formatter().parse()`. Replaced with cases that hit
distinct branches of the upstream validator. Test 4 (`{{var}}` as
literal) was dropped to avoid duplicating the side-by-side comparison
already in prompt-template-double-brackets-regression.spec.ts.
Mustache-mode invalid patterns (cases 6-9 in the issue) deferred to a
follow-up issue + PR.
This was referenced May 12, 2026
…ests The four bullets covering prompt-template-component-regression.spec.ts predated the per-test 1:1 convention adopted by Spec B (double-brackets) and Spec C (invalid-patterns). Replaced with six bullets that match the six @stable test() titles in Spec A, consistent with the format used by its sibling specs in the same section. Coverage Summary table regenerated via `npm run coverage:summary`: core-components total goes from 60 → 62 bullets, validated 52 → 54.
3 tasks
Address the review on PR #220: - Tighten the {1var} toast assertion from the bare digit "1" to the fragment "Invalid variables: 1" so the test cannot pass by matching incidental digit characters elsewhere in the toast — proves the upstream `_fix_variable` leading-digit branch fired. - Drop the three `(page as any).allowFlowErrors()` calls. The fixture only fails on `flow_error`-type events from /build/, /run/, or /events?event_delivery= (see tests/fixtures/fixtures.ts:230-262); HTTP 500s on /api/v1/validate/prompt are logged as `http_error` and never fail the test, so the opt-out was dead code and misleading. - Add a fourth rejection case `{var,name}` (comma) covering another character from the upstream `_INVALID_CHARACTERS` set. Common templating mistake (users writing `{a,b}` expecting a list). - Trim a duplicated rationale comment block in the spec — the doc Notes section already owns the explanation. Spec doc and QA-CHECKLIST updated to reflect the new test (6 total, 4 rejection + 2 positive). Coverage Summary and Phase 0 regenerated via `npm run coverage:summary` (96 -> 97 @stable tests). Helper extraction (also flagged in review) tracked as #222.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds regression coverage for the Prompt Template component’s f-string variable validation contract, ensuring invalid placeholder patterns surface backend ValueError details in the UI (toast) and do not create dynamic handles, while also capturing defensive “accepted” behaviors like {} and deduplication.
Changes:
- Added a new Playwright spec with
@stabletests covering invalid f-string patterns (dot/space/comma/leading digit) plus{}acceptance and variable deduplication. - Added new documentation for the invalid-patterns regression suite and updated the existing Prompt Template regression doc to reference it.
- Updated
QA-CHECKLIST.mdPrompt Template section and regenerated coverage summary/Phase 0 counts.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
tests/tests-automations/regression/core-components/prompt-template-invalid-patterns-regression.spec.ts |
New serial Playwright spec asserting rejection toasts + no dynamic handles for invalid f-string patterns, plus {} and dedup contracts. |
docs/core-components/prompt-template-invalid-patterns-regression.md |
New test documentation describing scope, steps, and upstream contracts for invalid pattern handling. |
docs/core-components/prompt-template-component-regression.md |
Updates “What this test does not cover” to point readers to the new invalid-patterns spec. |
QA-CHECKLIST.md |
Adds Prompt Template checklist bullets for the new cases and updates coverage summary counts/dates. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+91
to
+97
| // Runs the four-step rejection contract for a single invalid template: | ||
| // 1. submit the template via the prompt modal | ||
| // 2. assert the error toast carries the upstream ValueError title + detail | ||
| // + the offending variable name (so a stale buffer regression is caught) | ||
| // 3. assert no dynamic handle was created on the node | ||
| // 4. assert the modal stays in edit mode (frontend sets isEdit=true on error) | ||
| // |
| - `src/backend/base/langflow/api/v1/validate.py` — `POST /validate/prompt`: returns HTTP 500 with `detail=str(ValueError(...))` on rejection. The HTTP status and the `detail` shape are what the frontend's `onError` callback consumes. | ||
| - `src/frontend/src/modals/promptModal/index.tsx` — `genericModalBtnSave`, `modal-promptarea_prompt_template`, and the `usePostValidatePrompt` mutation's `onError` callback that maps the API error into the toast (title from `t("errors.prompt")`, detail from `error.response.data.detail`). Also sets `isEdit=true` so the modal stays open on error. | ||
| - `src/frontend/src/alerts/error/index.tsx` — `ErrorAlert` component renders with CSS class `.error-build-message`; the 5-second auto-dismiss timeout there defines the maximum window in which the toast assertion must run. | ||
| - `src/frontend/src/locales/en.json` — i18n key `errors.prompt` whose value `"There is something wrong with this prompt, please review it"` is asserted as a constant. Localizing this string would break tests 1–3. |
Comment on lines
+179
to
+184
| - [x] f-string parser rejects `{var.attr}` (dot notation) with an error toast and creates no handle → `core-components/prompt-template-invalid-patterns-regression.spec.ts` | ||
| - [x] f-string parser rejects `{var name}` (space inside identifier) with an error toast and creates no handle → `core-components/prompt-template-invalid-patterns-regression.spec.ts` | ||
| - [x] f-string parser rejects `{var,name}` (comma inside identifier) with an error toast and creates no handle → `core-components/prompt-template-invalid-patterns-regression.spec.ts` | ||
| - [x] f-string parser rejects `{1var}` (leading digit) with an error toast and creates no handle → `core-components/prompt-template-invalid-patterns-regression.spec.ts` | ||
| - [x] f-string parser accepts `{}` (empty braces) silently — no error, no handle → `core-components/prompt-template-invalid-patterns-regression.spec.ts` | ||
| - [x] f-string parser deduplicates repeated variables — `{name} and {name}` yields exactly one handle → `core-components/prompt-template-invalid-patterns-regression.spec.ts` |
…rns spec - Update runRejectionContract doc-comment to match its actual behavior (3 steps in helper; no-handle assertion at test body for expect-expect). - Correct External dependencies note on errors.prompt — toast title is asserted as a substring via toContainText, affecting all 4 rejection tests, not just 1–3. - Drop redundant Control+a before textarea.fill(). - Tighten preview.isVisible probe to 500ms (was 2000ms).
…mpt-template-invalid-patterns # Conflicts: # QA-CHECKLIST.md
Victor-w-Madeira
deleted the
feature/issue-214-prompt-template-invalid-patterns
branch
May 12, 2026 16:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
prompt-template-invalid-patterns-regression.spec.ts(6@stabletests) covering the f-string parser contract of the Prompt Template component:{var.attr}(dot),{var name}(space),{var,name}(comma),{1var}(leading digit): each surfaces the upstreamValueErroras an error toast (.error-build-message), creates no handle, and leaves the modal in edit mode so the user can correct the input.{}(empty braces filtered out byFormatter().parse());{name} and {name}(deduplicated to one handle).docs/core-components/prompt-template-invalid-patterns-regression.mdwith all mandatory sections.docs/core-components/prompt-template-component-regression.md— replaces the generic "Variable name validation" line under What this test does not cover with a pointer to the new spec.QA-CHECKLIST.md§3.2 with 6 new bullets (all[x]). Auto-generated Coverage Summary and Phase 0 block regenerated vianpm run coverage:summary.Scope adjustments vs. issue #214
After probing
POST /api/v1/validate/promptdirectly against Langflow 1.10.x, two of the originally proposed cases were dropped because the upstream parser does not reject them:{}— Python'sFormatter().parse()yields an emptyfield_namethatextract_input_variables_from_promptfilters out; save succeeds silently. Reframed as a positive-path defensive test.{var-name}— hyphen is not in_INVALID_CHARACTERS; the save succeeds and avar-namehandle is created. Replaced with{var name}(space),{var,name}(comma) and{1var}(leading digit), which exercise the_INVALID_CHARACTERSset check and the_fix_variableleading-digit branch respectively.Test 4 from the issue (
{{var}}as literal) was dropped to avoid duplicating an assertion already present inprompt-template-double-brackets-regression.spec.ts(test 2), where it sits naturally alongside the mustache-mode comparison.Mustache-mode invalid patterns (cases 6–9 in the issue) are deferred to a follow-up issue and a dedicated PR.
Test plan
npm run typecheckcleannpm run lint— no new errors (only the pre-existing(page as any)convention warnings used by sibling specs)npx playwright test prompt-template-invalid-patterns-regression.spec.ts→ 6/6 passing (~23 s on Langflow 1.10.x nightly)ERROR_TOAST_TITLEto a sentinel string; test failed at the expected assertion line, confirming the assertion is realnpm run coverage:summaryis idempotent (a second run produces no diff)Closes #214