Skip to content

chore(flow-functionality): add @stable and harden curlApiGeneration test - #139

Merged
daniellicnerski1 merged 3 commits into
mainfrom
validate/curl-api-generation
May 12, 2026
Merged

chore(flow-functionality): add @stable and harden curlApiGeneration test#139
daniellicnerski1 merged 3 commits into
mainfrom
validate/curl-api-generation

Conversation

@daniellicnerski1

Copy link
Copy Markdown
Collaborator

Summary

  • Replace the weak clipboardContent.length > 0 assertion with structural checks against the actual curl command (HTTP verb, /api/v1/run/<UUID>?stream=false, headers, JSON body keys).
  • Force the macOS/Linux platform sub-tab so the generated snippet is deterministic across machines (getOS() was driving the previous flakiness — Chromium's navigator.platform was producing the PowerShell variant).
  • Rename the test (curl_api_generation → sentence form), wrap each phase in test.step(), drop the misleading @api tag (this test does not exercise the REST API), add @stable.
  • Add the missing spec doc under docs/flow-functionality/curlApiGeneration.md.
  • Mark coverage as [x] in QA-CHECKLIST.md and QA-SCENARIOS-GUIDE.md scenario 5.1.

Validation

  • Baseline (pre-fix): 3/3 green — but copying the PowerShell variant; the old assertion was hiding it.
  • Forced-failure check: replaced "input_value": "Hello" with "ThisShouldFail" → test fails for the right reason.
  • Stability after fix: 3/3 green with --retries=0, average 3.7s per run.
  • npx eslint tests/tests-automations/regression/flow-functionality/curlApiGeneration.spec.ts — clean.
  • npx tsc --noEmit — clean.
  • No waitForTimeout, no if/else, no try/catch; 8 expects per test.

Test plan

  • weekly-stable.yml picks up the test on the next scheduled run via the new @stable tag.
  • CI pr-validation.yml (typecheck + lint) passes.

The previous version asserted only that the clipboard was non-empty, which
silently passed even when the wrong (PowerShell) variant was copied. The
test now explicitly switches to the macOS/Linux platform sub-tab and
validates the curl command structurally: HTTP verb, /api/v1/run/<UUID>
URL with stream flag, JSON content-type header, x-api-key header, and
the JSON body keys (input_value, session_id, output_type).

Adds the spec doc under docs/flow-functionality/, and marks coverage as
[x] in QA-CHECKLIST.md and QA-SCENARIOS-GUIDE.md scenario 5.1.

Drops the @api tag — this test exercises UI snippet generation, not the
REST API endpoint.
@Victor-w-Madeira
Victor-w-Madeira requested a review from Copilot May 11, 2026 17:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens the Playwright regression test that validates the “API Access → cURL” snippet generation/copy flow, and updates QA documentation to reflect that coverage.

Changes:

  • Strengthened curlApiGeneration e2e assertions from “clipboard non-empty” to structural validation of the generated macOS/Linux cURL command.
  • Reduced test flakiness by explicitly selecting the macOS/Linux platform variant and improving step-level traceability via test.step(), plus adding @stable.
  • Added/updated QA documentation to mark the scenario/checklist as covered and to document the spec for ongoing maintenance.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
tests/tests-automations/regression/flow-functionality/curlApiGeneration.spec.ts Makes the cURL generation test deterministic (forces macOS/Linux) and validates key command structure instead of only clipboard length.
QA-SCENARIOS-GUIDE.md Marks scenario 5.1 as covered.
QA-CHECKLIST.md Marks “Generate curl for API execution” as covered and links to the spec.
docs/flow-functionality/curlApiGeneration.md Adds a dedicated spec doc describing intent, validation criteria, and upstream dependencies.
Comments suppressed due to low confidence (1)

QA-SCENARIOS-GUIDE.md:310

  • Marking scenario 5.1 as covered is potentially misleading because the scenario’s step-by-step currently calls out verifying a curl -X POST method, while the actual generator/test expectation in flow-functionality/curlApiGeneration.spec.ts is curl --request POST. Consider updating the scenario text to assert the HTTP verb is POST (without prescribing -X vs --request) so the checklist/guide aligns with the test and the UI output format.
### 5.1 Generate curl for execution `[x]`

**Objective:** Verify that Langflow generates a valid `curl` command for flow execution.

**Step by step:**

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Daniel Licnerski Borges added 2 commits May 12, 2026 15:32
Replace prescriptive 'curl -X POST' wording with a method-agnostic
description, matching how the test actually validates (POST verb via
either '-X POST' or '--request POST'). Resolves Copilot review feedback
on PR #139.
@daniellicnerski1

Copy link
Copy Markdown
Collaborator Author

Copilot review — addressed

The single low-confidence note pointed out a real discrepancy: QA-SCENARIOS-GUIDE.md §5.1 step 4 was prescribing curl -X POST, while the actual snippet generator and the test assertion use curl --request POST. Fixed in e779583 by switching the scenario text to assert the HTTP method is POST without pinning the long/short flag form, so the guide tracks whichever variant the Langflow snippet generator emits.

Branch was also merged with the latest main (now 204 commits ahead — all from upstream main, plus one new commit for the doc fix). CI should re-run.

Full revalidation after the change

Step Result
npm run typecheck clean
npx eslint curlApiGeneration.spec.ts clean (0 errors / 0 warnings)
Static checklist (imports, false-positives, selectors, structure) pass
playwright test … --workers=1 --retries=0 1 passed (6.0s)
Force-fail (broke ^curl --request POST^curl --xyz-impossivel) failed exactly on the modified assertion, then reverted and re-ran clean
--trace=on trace.zip generated, steps coherent (7.8s)
🚨 Backend Error audit zero occurrences

@daniellicnerski1
daniellicnerski1 merged commit 2d16c31 into main May 12, 2026
2 checks passed
@Victor-w-Madeira
Victor-w-Madeira deleted the validate/curl-api-generation branch May 15, 2026 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants