Skip to content

feat(bundles): metapackage split (Phase A) — engine-only lfx, lfx-bundles long tail, 5 graduated partner packages - #13563

Merged
erichare merged 65 commits into
release-1.11.0from
bundles/foundation-discovery
Jun 23, 2026
Merged

feat(bundles): metapackage split (Phase A) — engine-only lfx, lfx-bundles long tail, 5 graduated partner packages#13563
erichare merged 65 commits into
release-1.11.0from
bundles/foundation-discovery

Conversation

@erichare

@erichare erichare commented Jun 9, 2026

Copy link
Copy Markdown
Member

Bundle Separation Phase A — metapackage split (1.11) · meta PR

This is the single PR for the entire Phase A bundle-separation epic. The original 10-PR stack has been fully collapsed into this branch as each ticket passed review — #13564, #13565, #13566, #13567, and #13568 merged here directly, and #13568 itself carried #13573, #13576, #13577, #13578, #13579, and #13580. Only this PR remains to merge into release-1.11.0.

Includes Oracle (#13502): the lfx-oracle bundle now ships in this branch as a sixth graduated package following the same conventions (manifest-shipping, bounded ranges, .dev0 lfx floor, migration-table entries, pilot-upgrade test). An earlier plan to land Oracle as a separate follow-up after this PR was changed — it is folded in here, and the lockfile, Docker images, and workspace members reflect that.

What it does: lfx becomes engine-only. The ~45-provider long tail moves into one manifest-less lfx-bundles metapackage discovered via a new lfx.bundles entry-point group, and five partner/flagship providers graduate to standalone manifest-shipping packages: lfx-openai, lfx-anthropic, lfx-amazon, lfx-datastax, lfx-cohere. Zero user-facing change: ext:<provider>:<Class>@official component ids are unchanged, in-tree shims keep from lfx.components.<provider> import X working, and existing flows load and run identically.

~328 files changed (+39k / −59k with rename detection; most of the delta is providers relocating from src/lfx/src/lfx/components/ into src/bundles/).

Part 1 — manifest-less lfx.bundles discovery + precedence tier (foundation)

  • New load_lfx_bundles_extensions (src/lfx/src/lfx/extension/loader/_bundles_root.py) iterates the lfx.bundles setuptools entry-point group, resolves each declared package via find_spec, folder-walks its immediate subdirectories, and registers each as a bundle at the @official slot with no extension.json (the langchain-community model). Reuses _load_bundle_directory, mirrors discover_inline_bundles.
  • Discovery precedence is now installed > seed > lfx_bundles > dev > inline. Manifest always wins: a manifest-shipping lfx-<provider> shadows the same-named metapackage provider with a typed bundle-shadowed warning, and the shadowed provider is never imported — this is what lets a provider graduate with no lockstep release.
  • New bundle-discovery-malformed error code, emitted on warnings (never flips ok / aborts startup), with per-mode codes for unresolvable declarations, plain-module targets, namespace-package portions, and invalid provider folder names. Manifest-less bundles get a typed reload refusal instead of a silent no-op.

Part 2 — lfx-bundles metapackage skeleton (#13564) + 45-provider move (#13568)

  • src/bundles/lfx-bundles/: single pyproject.toml declaring the lfx.bundles entry-point, a pre-release-safe lfx>=1.11.0.dev0,<2.0.0 floor, a generated all extra, and a README documenting the model + install stories. Hyphen dir name so release.yml's src/bundles/*/pyproject.toml glob builds it with zero workflow change.
  • 45 long-tail providers (chroma, groq, ollama, mistral, qdrant, tavily, mem0, …) move from src/lfx/src/lfx/components/ into src/bundles/lfx-bundles/src/lfx_bundles/, each discovered manifest-less at @official. In-tree marker shims preserve the old import paths; the migration table and component_index.json are updated append-only/surgically with sha256 re-verified. This set includes notdiamond and vlmrun, which an earlier revision of this branch had dropped without a bundle or shim; both are now relocated like the rest (bundle package + shim + migration-table targets + restored locale strings) so saved flows using the NotDiamond Router or VLM Run Transcription nodes keep loading unchanged.
  • Removes the orphaned scripts/ci/update_bundle_versions.py (+ tests) — the nightly bundle-rename track it served was retired by the canonical pre-release cutover (src/bundles/NIGHTLY.md).

Part 3 — engine-only lfx + lfx[bundles] extra (#13565)

  • pip install lfx → engine only; pip install "lfx[bundles]" → engine + the lfx-bundles long tail. Intentionally no lfx[all]. lfx-bundles appears only under extra == "bundles", never in core Requires-Dist.

Part 4 — partner graduations (#13573)

  • lfx-openai, lfx-anthropic, lfx-amazon, lfx-datastax, lfx-cohere: each ships extension.json (lfx.compat: ["1"]), a langflow.extensions entry-point, an lfx pin, and is a workspace member pinned by langflow. Partner set is disjoint from lfx-bundles (verified).
  • datastax (the heavy one): lfx.base.datastaxlfx_datastax.base, 74 backend tests move with it, 10 repo consumers rewritten (incl. the vector_store_rag starter project).
  • Validator fix surfaced by this work: validate_extension now accepts classes whose base is a derived Component base (LCVectorStoreComponent etc.) that inherit class-level outputs.
  • Lock diff adds the lfx-* package names (the engine split plus the graduated bundles — including lfx-bundles and lfx-oracle) and, for the Oracle bundle, the oracledb and langchain-oracledb third-party packages; no other third-party packages are added or removed.

Part 5 — CI: cross-bundle matrix (#13566) + components freeze gate (#13567)

  • .github/workflows/cross-bundle-test.yml: per (bundle × Python 3.10/3.13) — installs in-repo lfx + the bundle, imports the entry-point package, asserts lfx.bundles discovery is error-free, runs lfx extension validate for manifest bundles, and runs the bundle's own tests/. Discovers bundles via the same glob release.yml uses, so each new extraction self-registers. Triggers: pull_request, workflow_dispatch, weekly schedule, workflow_call.
  • scripts/ci/check_components_frozen.py + baseline frozen_component_dirs.txt: no new top-level provider directory may be added to src/lfx/src/lfx/components/ — new providers go to lfx-bundles or a graduated package. Removals are allowed (shim cleanup never trips it). Wired into extension-migration-checks.yml so it actually runs on PRs (re-homed from lint-py.yml during review).

Part 6 — hardening + review follow-ups (#13576#13580 and branch commits)

  • Bounded version ranges for the curated lfx-* packages (chore(bundles): bounded version ranges for curated lfx-* packages #13576); shim-contract lock tests + breakage audit (test(bundles): lock the in-tree shim contract + breakage audit #13577); bundle docs — install shapes, override rule, bundle_api_version (docs(bundles): install shapes, override rule, bundle_api_version #13578); stack-review fixes — symlink containment, idempotency, docs accuracy (fix(bundles): stack-review fixes — symlink containment, idempotency, docs accuracy #13579); tombstone for the broken legacy ZepChatMemory build method (fix(bundles): tombstone the broken legacy ZepChatMemory build method #13580).
  • Post-collapse review fixes on this branch: restore legacy-name resolution for ext components + starter-project sync; keep importable module paths in starter projects; restore loading.py / tableAutoCellRender to base-branch state (out-of-scope changes reverted); frontend test updates.
  • CI green-up (a05d937): the static→bundle move made two pieces of state order-dependent. (1) flatten_components_with_aliases now registers every component's identity aliases (registry key / ext class name / name) before any component's display_name alias, so a standalone component always wins its own type key over a Composio wrapper that merely shares its display label. A single setdefault pass previously let this resolve by registry load order, so update_starter_projects wrote the wrong metadata.module for the AgentQL node in News Aggregator.json / Price Deal Finder.json (red Test Starter Templates / test-starter-projects); both are corrected back to lfx.components.agentql.agentql_api.AgentQL, and an audit confirmed AgentQL and SerpAPI are the only such display-name collisions and both now resolve correctly. (2) SIDEBAR_BUNDLES name slugs for Notion and FAISS were still capitalized and so never matched the lowercase bundle category names — both bundles were silently dropped from the sidebar (red integration-side-bar Playwright e2e); lowercased to notion/faiss (display labels and icons unchanged). Order-independence regression tests added.

Test plan

  • Extension suite green (463 tests): discovery + @official registration, malformed-declaration / plain-module / namespace-portion / invalid-name warnings, manifest-shadows-manifestless, typed reload refusal
  • All five graduated packages register via the installed-manifest tier (20 components); manifest-less lfx_bundles registers its providers, disjoint from the partner set
  • 60 pilot-upgrade migration tests + 74 moved datastax tests pass; all shims import; engine-safe import lfx.components
  • lfx extension validate passes for all manifest bundles; component_index.json surgically updated, sha256 verified
  • uv sync resolves; wheels build; lfx METADATA carries lfx-bundles only under extra == "bundles"
  • Cross-bundle matrix green (all bundle × py3.10/3.13 legs); freeze gate passes on baseline, fails actionably on a simulated new provider dir
  • ruff + ruff format clean; mypy clean on new modules
  • Deterministic alias tiers (identity > display_name): AgentQL & SerpAPI resolve to the standalone component regardless of registry load order; 236/236 starter-projects template tests pass; bundles sidebar renders Notion + FAISS

Collapsed stack (all merged into this branch)

PR Scope
#13564 lfx-bundles metapackage skeleton
#13565 lfx[bundles] extra, engine-only lfx
#13566 cross-bundle CI matrix
#13567 lfx/components freeze gate
#13568 45-provider bulk move (carried the PRs below)
#13573 graduate openai / anthropic / amazon / datastax / cohere
#13576 bounded version ranges for curated lfx-*
#13577 shim contract tests + breakage audit
#13578 docs: install shapes, override rule, bundle_api_version
#13579 stack-review fixes (symlinks, idempotency, docs)
#13580 tombstone broken legacy ZepChatMemory build

Summary by CodeRabbit

  • New Features
    • Added manifest-less bundle discovery for distribution metapackages through entry-point registration
    • Immediate subdirectories are now automatically recognized as bundles without requiring configuration files
    • Updated bundle discovery precedence order: installed > seed > lfx_bundles > dev > inline
    • Added warning-only error handling for malformed bundle declarations that won't interrupt startup

… tier

Foundation for the bundle metapackage split (1.11). Adds a third
@official-slot discovery source: a distribution declaring the
[project.entry-points."lfx.bundles"] group ships a package whose immediate
subdirectories are each a manifest-less bundle, folder-walked and registered
at @official with no extension.json (the langchain-community model).

- new loader/_bundles_root.py::load_lfx_bundles_extensions, mirroring
  discover_inline_bundles but sourcing roots from the lfx.bundles entry-point
  group via find_spec and reusing _load_bundle_directory
- discovery precedence becomes installed > seed > lfx_bundles > dev > inline
  so a manifest-shipping lfx-<provider> always shadows the same-named
  provider in the metapackage (lets a provider graduate with no lockstep
  release), emitting the existing bundle-shadowed warning
- new bundle-discovery-malformed warning code, emitted on warnings (never
  flips ok / aborts startup) for unresolvable declarations and invalid
  provider folder names
- exported via loader/__init__ and the lfx.extension PEP-562 lazy surface

10 new tests in test_load_lfx_bundles.py; full extension unit suite (449)
passes; ruff + format clean; mypy clean on the new module.
@coderabbitai

coderabbitai Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: b3f7573a-5d52-48e4-ad6b-72a7c17913f4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Walkthrough

This PR adds manifest-less bundle discovery for the lfx.bundles entry-point group. The implementation resolves entry-point packages to filesystem directories, enumerates subdirectories as bundles registered at the @official slot, and integrates into the existing five-source discovery precedence with warning-only error handling for malformed declarations.

Changes

Manifest-less Bundle Discovery

Layer / File(s) Summary
Error codes and public API exposure
src/lfx/src/lfx/extension/errors.py, src/lfx/src/lfx/extension/__init__.py, src/lfx/src/lfx/extension/loader/__init__.py
New bundle-discovery-malformed warning-only error code with message template; load_lfx_bundles_extensions exported from lfx.extension and lfx.extension.loader via __all__ and lazy-loading mechanisms.
Core manifest-less bundle loading
src/lfx/src/lfx/extension/loader/_bundles_root.py
Resolves lfx.bundles entry-points to filesystem directories without importing, enumerates immediate subdirectories as bundle providers, validates names, handles duplicates via first-wins precedence, and emits warning-only bundle-discovery-malformed sentinels for resolution failures and invalid directory names.
Bundle discovery precedence and shadowing
src/lfx/src/lfx/interface/components.py
Updates _DISCOVERY_PRECEDENCE to include lfx_bundles between seed and dev; extends _resolve_bundle_shadowing to a five-source deduplication tuple; calls load_lfx_bundles_extensions() in import_extension_components and populates the registry with deduplicated results.
Unit tests and error snapshots
src/lfx/tests/unit/extension/loader/test_load_lfx_bundles.py, src/lfx/tests/unit/extension/test_errors.py
Tests bundle discovery, provider enumeration, entry-point resolution with fallback error handling, cross-source precedence/shadowing with installer-manifest and inline-source collisions, and error formatting snapshots for bundle-discovery-malformed.
API changelog documentation
BUNDLE_API.md
Documents manifest-less lfx.bundles discovery, load_lfx_bundles_extensions export, five-source precedence, warning-only error code, and metapackage PEP 508 compatibility gating.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Suggested labels

lgtm

Suggested reviewers

  • dkaushik94
  • ogabrielluiz
  • Cristhianzl
🚥 Pre-merge checks | ✅ 8 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Title check ⚠️ Warning The PR title mentions 'metapackage split (Phase A)' and 'engine-only lfx, lfx-bundles long tail, 5 graduated partner packages' but the actual changeset focuses narrowly on implementing manifest-less lfx.bundles discovery and precedence tier—not a full metapackage split or 5 graduated packages. Revise the title to accurately reflect the core change: 'feat(extension): manifest-less lfx.bundles discovery and precedence tier' or similar, as this PR is Phase A foundation work that enables the broader metapackage split rather than implementing it.
✅ Passed checks (8 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed Docstring coverage is 91.67% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Test Coverage For New Implementations ✅ Passed PR includes 10 comprehensive unit tests for new load_lfx_bundles_extensions covering discovery, registration, error handling, shadowing, and precedence with 34+ assertions and no placeholder tests.
Test Quality And Coverage ✅ Passed 10 tests, 34 assertions. Comprehensive pytest patterns. Validates bundle registration, malformed handling, deduplication, precedence - not smoke tests.
Test File Naming And Structure ✅ Passed test_load_lfx_bundles.py follows pytest backend convention: test_*.py name, 10 descriptive tests with docstrings, uses fixtures, helper functions, covers positive/negative/edge cases comprehensively.
Excessive Mock Usage Warning ✅ Passed Test file uses zero mock objects from unittest.mock or pytest-mock. Tests use simple test doubles and real file construction with appropriate design patterns—no excessive mocking detected.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch bundles/foundation-discovery

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Jun 9, 2026
erichare added 2 commits June 10, 2026 11:07
The BUNDLE_API.md changelog gate diffs each PR's branch against main, so
the entry covering this PR's surface additions (lfx.bundles discovery,
the lfx_bundles precedence tier, bundle-discovery-malformed) must live
on this branch, not only further up the stack. Text is verbatim from the
graduate-partners commit so the stacked merge dedupes trivially.
@github-actions github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Jun 10, 2026
@erichare
erichare requested a review from dkaushik94 June 10, 2026 20:39
@erichare
erichare marked this pull request as ready for review June 10, 2026 20:40
@github-actions github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Jun 10, 2026
@github-actions

github-actions Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

✅ Test Coverage Advisor

No source changes detected without accompanying tests. Thanks for keeping coverage up! 🎉

Advisory check only — never blocks merge.

@erichare
erichare requested a review from ogabrielluiz June 10, 2026 20:43
@github-actions github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Jun 10, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@BUNDLE_API.md`:
- Around line 405-421: Update the BUNDLE_API.md wording for manifest-less
lfx.bundles to avoid implying the CLI can validate those directories: change
“exempt from `lfx extension validate`” to something like “not validated / not
intended as input to `lfx extension validate`” and add a one-line explanation
that validate_extension() (the CLI check) requires an extension.json or
[tool.langflow.extension] in pyproject.toml so pointing the validator at a
manifest-less provider will report “manifest-not-found”; reference the loader
module src/lfx/src/lfx/extension/loader/_bundles_root.py and the
validate_extension() CLI behavior in the note.

In `@src/lfx/src/lfx/extension/loader/_bundles_root.py`:
- Around line 246-260: _spec_package_dir currently treats plain module specs as
package roots by returning Path(spec.origin).parent when
submodule_search_locations is absent; change it to reject non-package
entry-point targets by removing the fallback that returns the origin's parent.
Update _spec_package_dir so it only returns a Path when
spec.submodule_search_locations is present (i.e., a package or namespace
package) and otherwise returns None; keep the initial spec is None guard and do
not treat spec.origin as a package root.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: adefeccc-6f5e-4660-b680-b1263385f836

📥 Commits

Reviewing files that changed from the base of the PR and between 29140c0 and 5bec743.

📒 Files selected for processing (8)
  • BUNDLE_API.md
  • src/lfx/src/lfx/extension/__init__.py
  • src/lfx/src/lfx/extension/errors.py
  • src/lfx/src/lfx/extension/loader/__init__.py
  • src/lfx/src/lfx/extension/loader/_bundles_root.py
  • src/lfx/src/lfx/interface/components.py
  • src/lfx/tests/unit/extension/loader/test_load_lfx_bundles.py
  • src/lfx/tests/unit/extension/test_errors.py

Comment thread BUNDLE_API.md Outdated
Comment thread src/lfx/src/lfx/extension/loader/_bundles_root.py Outdated
…date wording

CodeRabbit review fixes on #13563:
- _spec_package_dir no longer falls back to a plain module's parent
  directory -- a single-file entry-point target now emits the
  bundle-discovery-malformed warning instead of folder-walking unrelated
  sibling directories as bundles (+ regression test)
- BUNDLE_API.md / loader docstring no longer say manifest-less providers
  are 'exempt from lfx extension validate'; they are not valid validator
  input (validate requires a manifest and reports manifest-not-found)
@github-actions github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Jun 10, 2026

@ogabrielluiz ogabrielluiz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @erichare, re-reviewed after the head moved. The six _bundles_root.py / errors.py notes from my last pass are all addressed: the dotted-find_spec parent import now catches broadly, the same-tier duplicate uses duplicate-lfx-bundles-provider instead of the cross-source bundle-shadowed template, the claimed-bundle skip-before-import avoids overwriting the winner's _lfx_ext.official.<bundle>.* modules, _spec_package_dirs walks every namespace portion with resolved-path dedup, and the plain-module case returns []. The two freeze-gate notes are covered too, the gate now runs on pull_request in extension-migration-checks.yml. Nice cleanup.

Two small things left, both docs:

The errors.py comment says the new lfx.bundles codes are "All warning-only" and "mirror the inline tier's split", but inline-bundle-name-invalid and inline-path-unreadable go on result.errors (so ok=False), while bundles-provider-name-invalid and bundles-root-unreadable go on .warnings (so ok=True). Only duplicate-inline-bundle actually matches. The behavior is fine, but the comment overclaims the mirror. Could you tighten it to say only the duplicate case mirrors, or move the two name/readable codes to .errors to match inline?

Manifest-less lfx.bundles providers still don't show up in lfx extension list. discover_all_extensions only walks installed + seed (both manifest-only) and never calls load_lfx_bundles_extensions, which is startup-only, so a live metapackage-served provider is invisible at list-time and the shadow only resolves at server startup. That's a reasonable design given the sys.modules namespace is assembled at startup, but BUNDLE_API.md's manifest-less block doesn't mention the list-time limitation. Could you add a line so the precedence story isn't read as list-time too?

"Groq": {"langchain-groq"},
"IBM watsonx.ai": {"langchain-ibm"},
"Ollama": {"langchain-ollama"},
"OpenAI": {"langchain-openai"},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @erichare, I think the preflight and the shim send users to two different packages here. On an engine-only lfx install, a flow with an OpenAI component goes like this: lfx run preflight hits this fallback and says pip install langchain-openai, the user installs it and re-runs, then graph load fails on the import shim with pip install lfx-openai. Two round-trips, and langchain-openai alone never fixes it because the missing piece is the component package, not the SDK.

Could the preflight name the lfx-<provider> / lfx-bundles distribution that's actually missing, since that's the engine-only case it's meant to protect? Or is the shim error meant to be the authoritative one and this is just a best-effort SDK hint?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dug into this before changing it, and I think the SDK is actually the right target for this table — flipping it to the bundle would send the wrong package on the path this table serves.

_PROVIDER_PACKAGE_FALLBACKS is consulted only via _detect_providers_from_template, i.e. the unified model / agent_llm / embeddings_model provider dropdowns. At runtime that path is:

LanguageModelComponent.build_modelget_llmget_model_class("ChatOpenAI")class_registry._MODEL_CLASS_IMPORTS["ChatOpenAI"] = ("langchain_openai", "ChatOpenAI", None)

so it imports langchain_openai directly and never touches lfx.components.openai — the shim doesn't fire on this path, and pip install langchain-openai makes the flow run in one step. The same table also feeds the wxO deployment artifact (build_langflow_artifact_bytes), whose runner builds models the same way, so naming the bundle there would drag in the whole bundle when only the SDK is needed.

The two-round-trip / shim-firing case you saw sounds like a standalone provider component loaded by module reference — a different path (static code analysis), which also already resolves to the SDK. If you've got that flow handy I'm happy to dig into that path separately, since it's a real wart even if it isn't this table. So I've left the table as-is for now; let me know if you'd still prefer the bundle name and I'll reconcile it with the wxO export expectations.

The three doc points are fixed in f516062: this comment's inline-mirror overclaim in errors.py, the lfx extension list list-time limitation in BUNDLE_API.md, and the lfx-bundles README's "empty skeleton" framing.

Comment thread src/bundles/lfx-bundles/README.md Outdated
… metapackage README

Addresses three documentation nits from the PR #13563 re-review:

- errors.py: the lfx.bundles diagnostics comment claimed a full mirror of the
  inline tier's split, but only the duplicate case matches. name-invalid and
  root-unreadable are warning-only here where their inline counterparts are
  hard errors (ok=False). Tighten the comment to match actual behavior.
- BUNDLE_API.md: note that manifest-less lfx.bundles providers are invisible to
  `lfx extension list` (discover_all_extensions walks only installed + seed;
  load_lfx_bundles_extensions runs in the startup component-loading path), so
  the precedence story isn't read as a list-time property.
- lfx-bundles/README.md: the bulk move has landed on this branch (71 providers,
  populated per-provider extras, generated `all`), so drop the "empty skeleton /
  available later" framing and document the final state.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ogabrielluiz

Copy link
Copy Markdown
Contributor

Ran a probe on this branch with engine-only lfx (no bundles), loading the stock 1.10 starters, and there's a gap worth raising on the error path.

The actionable install guidance only exists on two routes: the legacy import shim, and the lfx run preflight (run/base.py:324). The server run path has neither. simple_run_flow -> run_graph_internal -> graph.arun() never calls find_missing_dependencies (a grep of src/backend for it is empty), and the build layer passes the raw string straight through (format_exception_message in api/utils/core.py only special-cases SyntaxError). So a flow loaded through the UI or API that needs a plain SDK dep surfaces as:

Error creating class. ModuleNotFoundError(No module named 'langchain_chroma').

Repro: load Vector Store RAG.json from v1.10.0 on this branch with engine-only lfx. It uses core Knowledge components, but instantiating them imports langchain_chroma, which a bare lfx doesn't carry. lfx run catches this and says pip install chromadb langchain-chroma. The UI/API path gives the cryptic message above with no guidance.

The graduated providers are fine, for what it's worth. A flow importing lfx.components.datastax.* keeps the shim message through the wrap (Error creating class. ModuleNotFoundError(The 'datastax' components moved to the 'lfx-datastax' distribution...)), so that case stays actionable. It's the plain SDK deps on the server path that don't.

Could we catch ModuleNotFoundError at vertex build and attach the same install guidance, or run the preflight on the server run path too? Right now the helpful message only reaches CLI users.

…rver run path

A flow loaded through the UI/API on an engine-only `lfx` (no bundles)
failed deep in the vertex build with a bare
`Error creating class. ModuleNotFoundError(No module named 'langchain_chroma')`.
The actionable install guidance only existed on the `lfx run` preflight
and the legacy import shim; the server run path
(simple_run_flow -> run_graph_internal -> graph.arun()) had neither.

Both vertex-build error formatters (lfx's `format_exception_message`, used
by graph.arun(), and langflow's, used by the build/chat endpoints) now
detect a plain ModuleNotFoundError in the cause chain and attach the same
`pip install <package>` guidance the CLI gives, mapping the missing import
to its PyPI distribution via the existing requirements resolver. Bundle-shim
"components moved to ..." messages are left untouched (their message does
not start with "No module named"), so graduated providers stay actionable.
@erichare

Copy link
Copy Markdown
Member Author

Good catch — fixed in 1e2395a.

Went with option 1 (catch ModuleNotFoundError at vertex build and attach guidance) since it covers both server surfaces in one place: the UI build endpoint (build.py) and the graph.arun() run path each format vertex-build failures through a format_exception_message, so I wired the hint into both. When the cause chain bottoms out in a plain No module named '...', we now map the import to its PyPI distribution (reusing the requirements analyzer's resolver) and return the same pip install <package> guidance the lfx run preflight gives.

Vector Store RAG.json on engine-only lfx now surfaces:

No module named 'langchain_chroma'. This flow needs a Python package that is not installed in this environment.

Install it and re-run:
  pip install langchain-chroma

Provider components moved out of the lfx engine in the bundle split, so a flow can require packages that a bare `pip install lfx` does not include.

The graduated-provider shim messages are deliberately left untouched — their str() doesn't start with No module named, so the "components moved to the 'lfx-datastax' distribution" text still flows through unchanged. Added unit coverage for both the plain-SDK and shim cases.

@ogabrielluiz

Copy link
Copy Markdown
Contributor

Verified on 1e2395a, the SDK case is fixed. Loading Vector Store RAG.json from v1.10.0 engine-only now surfaces pip install langchain-chroma through the formatter instead of the bare Error creating class. ModuleNotFoundError(...). Nice.

One thing on the graduated case. The commit says the shim "components moved to ..." messages are left untouched, but get_causing_exception walks to the deepest __cause__, and the shim raises its curated message from the original error, so the chain bottoms out at the raw No module named:

[0] ValueError: Error creating class...
[1] ModuleNotFoundError: The 'datastax' components moved to 'lfx-datastax'... (or pip install langflow)
[2] ModuleNotFoundError: No module named 'lfx_datastax'

It unwraps past [1] to [2], so the hint fires and the message gets regenerated as pip install lfx-datastax. Right package, but the curated shim text (including the "or pip install langflow, which bundles it" part) is dropped on this path rather than passed through. Could get_causing_exception stop at the first ModuleNotFoundError whose message doesn't start with "No module named", so the curated shim message wins when there is one?

@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Build successful! ✅
Deploying docs draft.
Deploy successful! View draft

…ption formatter

get_causing_exception walked to the deepest __cause__, so a graduated
provider's curated "components moved to ..." ModuleNotFoundError (raised
`from` the raw "No module named 'lfx_<x>'") was unwrapped past it. The
formatter then regenerated bare "pip install lfx-<x>" guidance, dropping
the curated text — including the "or pip install langflow, which bundles
it" clause.

Both get_causing_exception copies (lfx + langflow) now stop at the first
ModuleNotFoundError whose message is not a raw "No module named ..." (the
curated shim error), and module_not_found_hint surfaces that curated
message verbatim. Plain SDK import errors are unchanged.

Adds regression tests reproducing the real 3-level cause chain on both
formatters; the old test used a truncated chain that hid this.
@erichare

Copy link
Copy Markdown
Member Author

@ogabrielluiz good catch — done in f7b9cb3, exactly your suggestion.

get_causing_exception (both the lfx and langflow copies) now stops at the first ModuleNotFoundError whose message isn't a raw No module named ... — i.e. the curated shim error [1] — so it no longer unwraps past it to [2]. module_not_found_hint then surfaces that curated message verbatim, so the full "moved to 'lfx-datastax'... / or pip install langflow, which bundles it" text wins. Plain SDK errors (the Chroma path you verified) are unchanged.

Also added regression tests with the real 3-level chain on both formatters — the old shim test used a truncated chain (curated error with no raw cause underneath), which is why it was green and hid this.

@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Build successful! ✅
Deploying docs draft.
Deploy successful! View draft

erichare and others added 4 commits June 18, 2026 16:11
…s detection (#13732)

fix(bundles): cover OpenRouter/IBM WatsonX in wxO requirements detection

The bundle split removed the langchain provider SDKs from lfx's own
dependency tree, which makes flow→requirements detection load-bearing for
Watsonx Orchestrate deployments: the runner installs only `lfx` + the
packages `generate_requirements_from_flow` emits, so a provider missing
from the resolution tables yields an empty requirements set and the
deployed flow fails to import its model class.

Two unified-selectable providers were unresolved:
- OpenRouter (runs on ChatOpenAI) was in no resolution table -> [].
- "IBM WatsonX" (catalog casing) only matched the legacy "IBM watsonx.ai"
  key; langchain-ibm landed only by an embedding-path coincidence.

Add both to _PROVIDER_PACKAGE_FALLBACKS (keyed by the exact catalog
strings) so every provider in MODEL_PROVIDER_METADATA resolves.

Tests:
- TestBundleSeparationOrchestrate in test_flow_requirements.py: catalog
  coverage guard, OpenRouter/IBM regressions, and split-contract guards
  (bundle dists and langchain imports must stay out of lfx's tree).
- End-to-end artifact tests in test_watsonx_orchestrate.py asserting a
  bundle-provider flow emits the langchain SDK (not lfx-openai/lfx-bundles)
  and that requirements.txt pins lfx as its first line.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Build successful! ✅
Deploying docs draft.
Deploy successful! View draft

…-of-war

The datastax bundle's pyproject.toml declares a [tool.ruff.lint.per-file-ignores]
table, which makes ruff treat the directory as a standalone config root. It never
restated line-length, so ruff fell back to its default of 88 there. As a result
`make format_backend` (per-directory config discovery) wrapped lines to 88 while
the pre-commit ruff-format hook (run with --config pyproject.toml, forcing the
repo-wide 120) unwrapped them back, leaving the two formatters fighting forever.

Restate line-length = 120 so both config-resolution paths agree. Use the explicit
line-length rather than extend=../../../pyproject.toml to avoid pulling in the
root lint ruleset, which the bundle intentionally does not enforce.
@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Build successful! ✅
Deploying docs draft.
Deploy successful! View draft

1 similar comment
@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Build successful! ✅
Deploying docs draft.
Deploy successful! View draft

@github-actions

github-actions Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Build successful! ✅
Deploying docs draft.
Deploy successful! View draft

@ogabrielluiz

Copy link
Copy Markdown
Contributor

Hey @erichare, quick one on the precedence chain. LE-1491 lists langflow.plugins as a component tier below lfx.bundles, but as far as I can tell langflow.plugins is only read in plugin_routes.py for route registrars now. Nothing loads its component entry-points into the palette, so there's no lfx.bundles-vs-plugins collision to resolve, and filter_component_entry_points just keeps a manifest dist's component EP from being called as a route registrar. Is langflow.plugins still meant to be a component source somewhere I'm missing? I don't think it's a problem, just want to make sure the spec tier isn't pointing at something real we skipped.

@erichare

Copy link
Copy Markdown
Member Author

Hey @erichare, quick one on the precedence chain. LE-1491 lists langflow.plugins as a component tier below lfx.bundles, but as far as I can tell langflow.plugins is only read in plugin_routes.py for route registrars now. Nothing loads its component entry-points into the palette, so there's no lfx.bundles-vs-plugins collision to resolve, and filter_component_entry_points just keeps a manifest dist's component EP from being called as a route registrar. Is langflow.plugins still meant to be a component source somewhere I'm missing? I don't think it's a problem, just want to make sure the spec tier isn't pointing at something real we skipped.

No, you're not missing a component source. The LE-1491 tier is describing a precedence relationship (manifest is source-of-truth over a dist's legacy langflow.plugins component EPs), but the losing side of that relationship has no live palette consumer — so there's genuinely no lfx.bundles-vs-plugins collision to resolve at the component level. The only residual handling is the plugin_routes guard, which is doing the conservative thing.

erichare added 2 commits June 23, 2026 10:52
Both providers were dropped during the bulk long-tail move with no bundle
and no compatibility shim, removing them from the registry,
component_index, migration table and locales. That breaks saved flows
using the NotDiamond Router or VLM Run Transcription nodes (vlmrun ships
in langflow[complete]), contradicting the zero-user-facing-change goal.

Relocate both the same way as the other long-tail providers:
- move implementations into src/bundles/lfx-bundles/src/lfx_bundles/
- restore lfx.components.{notdiamond,vlmrun} marker shims
- re-register both in lfx/components/__init__.py
- add per-provider extras (vlmrun -> vlmrun[all]) and the 'all' aggregate
- add migration_table targets ext:{notdiamond,vlmrun}:<Class>@official
- restore the 32 en.json strings; refresh uv.lock
The preflight is wired into 'lfx run'; 'lfx serve' has its own
module-not-found hint at request time and does not call it.

@ogabrielluiz ogabrielluiz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving on the DX and backward-compat front. Everything I raised is resolved and I verified the fixes by running them on the branch, not just reading the commits:

  • Missing-SDK deps now surface pip install <pkg> guidance on the server/UI run path, not just lfx run (1e2395a).
  • Graduated-provider errors keep the curated "moved to lfx-" shim message, while plain SDK deps get regenerated guidance (f7b9cb3).
  • Docs fixes: README install state, the error-tier mirror claim, the list-time discovery note (f516062).

Stock 1.10/1.9 flows on core components load unchanged engine-only, and the langflow.plugins tier question checked out as a precedence description with no live palette consumer, so nothing real is being skipped there.

Scope note: this was the DX and user-confusion lens plus the discovery/precedence, shim/migration, and preflight areas. I didn't line-by-line audit all 573 files for correctness, so this approval is on that scope.

@github-actions

github-actions Bot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Build successful! ✅
Deploying docs draft.
Deploy successful! View draft

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants