fix(benchmarks): use TCP probe for langflow_run_http_ready readiness

[ogabrielluiz]

Summary

Swaps the langflow_run_http_ready benchmark scenario from scraping uvicorn's Application startup complete. stdout marker to a TCP connect probe against 127.0.0.1:7860. The scenario was previously sentinel (mean_ms: 0, runs: 0) because the structlog processor pipeline swallows the marker line the supervisor was waiting for. TCP connect is a ground-truth readiness signal that does not depend on logging at all — the sibling _langflow_no_change_restart_supervisor.py already uses this approach and its own header note called out this exact class of failure.

What changed

• src/backend/tests/benchmarks/scenarios/_langflow_supervisor.py: replaced the stderr line-scan loop with a socket.create_connection poll (50 ms interval), added a background thread that drains the child's merged stdout so CI logs still show boot progress, and kept the same exit-code semantics (0 ready, 2 timeout, 3 early exit).
• src/backend/tests/benchmarks/scenarios/langflow_run.py: updated the module docstring to reflect TCP-based readiness.
• .github/workflows/cold-start-benchmark.yml: dropped stale references to the "structlog marker issue," bumped the langflow_run_http_ready job timeout from 5 min to 10 min now that it actually runs to completion, and rewrote the continue-on-error comment to say this flag stays set only until a run-benchmark-snapshot captures a real baseline into thresholds.json. Also dropped the stale sentinel caveat from the generated _note.

Why continue-on-error stays

src/backend/tests/benchmarks/thresholds.json still carries { mean_ms: 0, runs: 0 } for this scenario. The driver's verify gate treats baseline_ms <= 0 as a sentinel-trip — any finite current mean would be flagged as an infinite regression. Once a run-benchmark-snapshot run swaps the 0 for a real number, a follow-up PR can drop langflow_run_http_ready from the continue-on-error expression. Comments in the workflow file describe this ordering.

Verification

• uv run ruff check and uv run ruff format --check — clean.
• cd src/backend && uv run pytest tests/benchmarks/tests/test_driver_verify.py — 4/4 pass.
• Smoke test: ran the supervisor against a stub TCP listener on a random port — detects readiness and emits LANGFLOW_READY_MS. Ran it against a child that exits early — correctly returns 3.
• CI cold-start-benchmark workflow on this branch will exercise the scenario end-to-end; numbers will surface in the matrix job output.

Stacked on

#12788 (cold-start/05-service-init-container).

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: e07dae1e-32a0-4716-8eaa-8de6b6ded1de

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch cold-start/fix-http-ready-tcp-probe

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[erichare]

LGTM! Minor comments below, but approving

Verdict: Approve. Small, well-scoped fix with solid rationale. Brings langflow_run_http_ready into alignment with the already-proven pattern in _langflow_no_change_restart_supervisor.py.

What it does

Replaces stderr line-scanning for uvicorn's Application startup complete. marker (swallowed by langflow's structlog pipeline → sentinel mean_ms: 0) with a TCP connect probe on 127.0.0.1:7860, plus a daemon thread draining child stdout so CI logs still show boot progress.

Strengths

• Pattern consistency: identical readiness strategy to the sibling _langflow_no_change_restart_supervisor.py, which even explicitly flagged this exact failure class in its header note.
• Exit-code semantics preserved (0/2/3) via an exit_code accumulator, with clearer stderr messages for each path.
• No deadlock risk: drain thread is daemon=True with join(timeout=2); stop_event is belt-and-suspenders — closing child stdout on SIGTERM unblocks the for line in stream loop anyway.
• Workflow comment updates are thorough: the 5→10min timeout bump is justified (tight budget only made sense while scenario was sentinel-failing), and the ordering — land fix → capture baseline via run-benchmark-snapshot → drop from continue-on-error in follow-up — is sensible.

Minor notes (LOW, non-blocking)

1. Port-already-in-use false positive: if anything else is bound to 127.0.0.1:7860, _tcp_ready returns true immediately and the supervisor reports a near-zero LANGFLOW_READY_MS while the child fails to bind. Pre-existing property of the sibling supervisor too; each matrix job runs on its own runner so not a practical concern, but worth remembering when someone tries to run two scenarios concurrently locally.
2. stdout buffering: Popen(text=True) without bufsize=1 means log lines may arrive in chunks. Purely a CI-log-visibility thing, not correctness.
3. exit_code = 0 initialization ([_langflow_supervisor.py:89](https://claude.ai/epitaxy/src/backend/tests/benchmarks/scenarios/_langflow_supervisor.py:89)) is redundant — all success paths set ready_at and bypass the return exit_code or 3. Harmless.

No CRITICAL/HIGH issues. Test plan in PR body (ruff, test_driver_verify.py 4/4, stub-listener smoke) is appropriate for the scope.

[github-actions]

Frontend Unit Test Coverage Report

Coverage Summary

Lines	Statements	Branches	Functions
	37.45% (44813/119641)	67.62% (6198/9165)	37.13% (1029/2771)

Unit Test Results

Tests	Skipped	Failures	Errors	Time
4241	0 💤	0 ❌	0 🔥	9m 32s ⏱️

[codecov]

Codecov Report

❌ Patch coverage is 81.81818% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.91%. Comparing base (1f894f7) to head (9d1ccf6).
⚠️ Report is 162 commits behind head on cold-start/01-measurement-foundation.

Files with missing lines	Patch %	Lines
src/backend/base/langflow/server.py	71.42%	2 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (1f894f7) and HEAD (9d1ccf6). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (1f894f7)	HEAD (9d1ccf6)
frontend	2	1

Additional details and impacted files

@@                            Coverage Diff                            @@
##           cold-start/01-measurement-foundation   #12798       +/-   ##
=========================================================================
- Coverage                                 52.78%   41.91%   -10.88%     
=========================================================================
  Files                                      2024     1953       -71     
  Lines                                    183804   180256     -3548     
  Branches                                  27344    10724    -16620     
=========================================================================
- Hits                                      97018    75547    -21471     
- Misses                                    85693   103542    +17849     
- Partials                                   1093     1167       +74     

Flag	Coverage Δ
backend	50.36% <71.42%> (-5.52%)	⬇️
frontend	37.45% <ø> (-15.36%)	⬇️
lfx	51.04% <100.00%> (+1.36%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/lfx/src/lfx/custom/validate.py	56.84% <100.00%> (+3.37%)	⬆️
src/backend/base/langflow/server.py	62.13% <71.42%> (+1.67%)	⬆️

... and 1366 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Cold Start Benchmark: regression detected (1 scenario(s) failed)

Baseline ref: 12798/merge@93c8aa1089ea04abeff5e5e1b14cc456460b9ac3 captured 2026-04-22 on ubuntu-latest (GitHub Actions). Allowed regression: 15%. Measurement mode: bytecode_compile_delta.

scenario	baseline_ms	current_ms	delta_pct	allowed_pct	status
lfx_with_flow_prebaked	8334.9	10117.3	+21.4%	15%	FAIL

Hyperfine JSON artifacts: see the cold-start-benchmark-reports workflow artifact. Local paths: reports/<scenario>.json.

Per to merge anyway, apply the benchmarks:override label AND document the justification in the PR description.

Measurement mode: bytecode_compile_delta

[github-actions]

Cold Start Benchmark: regression detected (1 scenario(s) failed)

Baseline ref: 12798/merge@93c8aa1089ea04abeff5e5e1b14cc456460b9ac3 captured 2026-04-22 on ubuntu-latest (GitHub Actions). Allowed regression: 15%. Measurement mode: bytecode_compile_delta.

scenario	baseline_ms	current_ms	delta_pct	allowed_pct	status
lfx_with_flow	15406.8	17994.8	+16.8%	15%	FAIL

Hyperfine JSON artifacts: see the cold-start-benchmark-reports workflow artifact. Local paths: reports/<scenario>.json.

Per to merge anyway, apply the benchmarks:override label AND document the justification in the PR description.

Measurement mode: bytecode_compile_delta

[github-actions]

Cold Start Benchmark: regression detected (1 scenario(s) failed)

Baseline ref: 12798/merge@93c8aa1089ea04abeff5e5e1b14cc456460b9ac3 captured 2026-04-22 on ubuntu-latest (GitHub Actions). Allowed regression: 15%. Measurement mode: bytecode_compile_delta.

scenario	baseline_ms	current_ms	delta_pct	allowed_pct	status
lfx_bare	9881.4	14961.4	+51.4%	15%	FAIL

Hyperfine JSON artifacts: see the cold-start-benchmark-reports workflow artifact. Local paths: reports/<scenario>.json.

Per to merge anyway, apply the benchmarks:override label AND document the justification in the PR description.

Measurement mode: bytecode_compile_delta

[github-actions]

Cold Start Benchmark: regression detected (1 scenario(s) failed)

Baseline ref: 12798/merge@93c8aa1089ea04abeff5e5e1b14cc456460b9ac3 captured 2026-04-22 on ubuntu-latest (GitHub Actions). Allowed regression: 15%. Measurement mode: bytecode_compile_delta.

scenario	baseline_ms	current_ms	delta_pct	allowed_pct	status
lfx_with_flow_prebaked	8334.9	9988.7	+19.8%	15%	FAIL

Hyperfine JSON artifacts: see the cold-start-benchmark-reports workflow artifact. Local paths: reports/<scenario>.json.

Per to merge anyway, apply the benchmarks:override label AND document the justification in the PR description.

Measurement mode: bytecode_compile_delta

[github-actions]

Cold Start Benchmark: regression detected (1 scenario(s) failed)

Baseline ref: 12798/merge@93c8aa1089ea04abeff5e5e1b14cc456460b9ac3 captured 2026-04-22 on ubuntu-latest (GitHub Actions). Allowed regression: 15%. Measurement mode: bytecode_compile_delta.

scenario	baseline_ms	current_ms	delta_pct	allowed_pct	status
lfx_bare	9881.4	15393.6	+55.8%	15%	FAIL

Hyperfine JSON artifacts: see the cold-start-benchmark-reports workflow artifact. Local paths: reports/<scenario>.json.

Per to merge anyway, apply the benchmarks:override label AND document the justification in the PR description.

Measurement mode: bytecode_compile_delta

[ogabrielluiz]

Consolidated into #12783 for QA. The chain merges that brought this branch's content up to the base PR caused GitHub to auto-mark this as merged; the actual landing happens when #12783 merges into release-1.10.0. Original review thread preserved here.