ci(bundles): cross-bundle test matrix (lfx contract axis) (#13566)

* ci(bundles): add cross-bundle test matrix (lfx contract axis)

Tests every extracted bundle (the lfx-bundles metapackage + each graduated
lfx-<provider>) against the lfx contract surface it depends on -- overdue
since 1.10 left 4 independently-versioned bundles depending on lfx.

- .github/workflows/cross-bundle-test.yml: discovers bundles via the same
  src/bundles/*/pyproject.toml glob release.yml uses, then per
  (bundle x python 3.10/3.13): installs the in-repo lfx + the bundle, imports
  the declared entry-point package, asserts lfx.bundles discovery is
  error-free (for the metapackage), runs `lfx extension validate` for
  manifest bundles, and runs the bundle's own tests/.
- Triggers: pull_request (src/bundles, src/lfx), workflow_dispatch, a weekly
  schedule, and workflow_call.

The lfx-minor axis seeds with the in-repo lfx (the 1.10 line is unpublished);
when minors publish, add the version dimension (oldest+latest get full tests,
every supported minor gets contract-smoke) per the epic's cost-control shape.

Verified: the workflow YAML parses and the contract-smoke logic imports a
real bundle (lfx_arxiv) cleanly.

* fix(ci): cross-bundle smoke tolerates extras-less SDK degradation

Review findings on the cross-bundle matrix:

- CRITICAL: the contract smoke asserted zero discovery errors, but the CI
  venv installs lfx-bundles WITHOUT per-provider extras, so providers whose
  modules import their SDK at top level degrade with module-import-failed --
  the expected graceful-degradation contract. The smoke now fails only on
  structural codes and reports the degraded-module count. (Graduated partner
  bundles carry their deps directly, not as extras, so their steps are
  unaffected.)
- the scheduled run now actually delivers the exhaustive grid the header
  promised (all supported Pythons on schedule; oldest+latest on PRs)
- concurrency group with cancel-in-progress on PRs so stacked bundle PRs
  don't queue N-bundles x N-pythons jobs per push

* ci(bundles): tomli fallback for the py3.10 cross-bundle smoke

tomllib is stdlib only from Python 3.11, so the contract-smoke heredoc
failed with ModuleNotFoundError on every py3.10 matrix leg. Install the
tomli backport into the smoke venv and import it as a fallback.

* fix(test): coherent sys.modules restore in the ibm without_ibm_db fixture

The cross-bundle matrix's py3.10 leg exposed test-ordering pollution in
the ibm bundle suite: without_ibm_db monkeypatch-deleted three lfx_ibm
modules and re-imported them mid-test, but entries created during the
test survive teardown (delitem with raising=False records nothing for
keys it didn't find) and re-imported parents never regain the submodule
attribute bindings that mock.patch target resolution walks on Python
3.10 (3.11+ mock resolves via sys.modules and tolerates the
incoherence). Two fixture uses in sequence left the tree inconsistent
and 29 later watsonx tests failed with AttributeError on the package.

Snapshot, drop, and restore the entire lfx_ibm module tree wholesale
instead. Full suite now passes on both 3.10 and 3.13.

Author: erichare

.github/workflows/cross-bundle-test.yml

@@ -0,0 +1,160 @@
+name: Cross-Bundle Test
+
+# Tests every extracted bundle (the lfx-bundles metapackage + each graduated
+# lfx-<provider>) against the lfx contract surface it depends on. lfx is the
+# PRIMARY axis because the BUNDLE_API contract lives in lfx: a bundle that
+# imports cleanly, validates, and passes its tests against a given lfx is
+# compatible with that lfx minor.
+#
+# Cost-control shape (from the bundle-separation epic):
+#   - contract-smoke (install + import + discover + validate): every bundle,
+#     on the oldest and latest supported Python.
+#   - bundle tests (pytest the bundle's own tests/): same matrix.
+#   - scheduled run: weekly, the exhaustive grid.
+#
+# The lfx-minor axis is currently a single entry -- the IN-REPO lfx -- because
+# the 1.10 line is not yet published to PyPI. When lfx minors publish, add an
+# explicit lfx-version matrix dimension here (oldest + latest get the full
+# tests; every supported minor gets contract-smoke), and wire langflow RCs as
+# the secondary axis via workflow_call from the release pipeline.
+
+on:
+  workflow_call:
+  workflow_dispatch:
+  pull_request:
+    paths:
+      - "src/bundles/**"
+      - "src/lfx/**"
+      - ".github/workflows/cross-bundle-test.yml"
+  schedule:
+    - cron: "0 6 * * 1" # Monday 06:00 UTC -- exhaustive grid
+
+# Stacked bundle PRs would otherwise queue N-bundles x N-pythons jobs per push.
+concurrency:
+  group: cross-bundle-${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  discover:
+    name: Discover bundles
+    runs-on: ubuntu-latest
+    outputs:
+      bundles: ${{ steps.find.outputs.bundles }}
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          persist-credentials: false
+      - id: find
+        name: Enumerate src/bundles/*/pyproject.toml
+        run: |
+          shopt -s nullglob
+          dirs=()
+          for p in src/bundles/*/pyproject.toml; do
+            dirs+=("$(dirname "$p")")
+          done
+          if [ ${#dirs[@]} -eq 0 ]; then
+            echo "No bundles found under src/bundles/*/" >&2
+            echo "bundles=[]" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+          json=$(printf '%s\n' "${dirs[@]}" | sort | jq -R . | jq -cs .)
+          echo "bundles=$json" >> "$GITHUB_OUTPUT"
+          echo "Discovered bundles: $json"
+
+  bundle:
+    name: ${{ matrix.bundle }} (py${{ matrix.python-version }})
+    needs: discover
+    if: needs.discover.outputs.bundles != '[]'
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        bundle: ${{ fromJson(needs.discover.outputs.bundles) }}
+        # PR runs: oldest + latest supported Python (cost-control smoke).
+        # Scheduled runs: every supported Python (the exhaustive grid).
+        # The lfx-minor axis collapses to the in-repo lfx until the 1.10 line
+        # publishes (see header).
+        python-version: ${{ github.event_name == 'schedule' && fromJson('["3.10", "3.11", "3.12", "3.13", "3.14"]') || fromJson('["3.10", "3.13"]') }}
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@v6
+        with:
+          enable-cache: true
+          cache-dependency-glob: "uv.lock"
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install in-repo lfx + the bundle into a clean venv
+        run: |
+          uv venv
+          # The in-repo lfx satisfies the bundle's `lfx>=X,<Y` pin and is the
+          # contract surface under test; pytest is for the bundle's own tests.
+          # tomli backports stdlib tomllib for the py3.10 matrix leg.
+          uv pip install ./src/lfx "./${{ matrix.bundle }}" pytest pytest-asyncio tomli
+
+      - name: Contract smoke -- import + discovery
+        run: |
+          .venv/bin/python - "${{ matrix.bundle }}" <<'PY'
+          import importlib
+          import sys
+          from pathlib import Path
+
+          try:
+              import tomllib  # stdlib from 3.11
+          except ModuleNotFoundError:
+              import tomli as tomllib  # py3.10 backport, installed above
+
+          bundle_dir = Path(sys.argv[1])
+          meta = tomllib.loads((bundle_dir / "pyproject.toml").read_text())
+          name = meta["project"]["name"]
+          eps = meta["project"].get("entry-points", {})
+
+          # Manifest bundles declare langflow.extensions; the manifest-less
+          # metapackage declares lfx.bundles. Either way the declared package
+          # must import.
+          modules = list(eps.get("langflow.extensions", {}).values()) or list(eps.get("lfx.bundles", {}).values())
+          assert modules, f"{name}: no langflow.extensions or lfx.bundles entry-point declared"
+          for value in modules:
+              importlib.import_module(value.split(":", 1)[0])
+              print(f"  imported {value}")
+
+          # The manifest-less metapackage must be discoverable by the loader.
+          # This venv installs the bundle WITHOUT its per-provider extras, so
+          # providers whose modules import their SDK at top level degrade with
+          # `module-import-failed` -- that is the expected graceful-degradation
+          # contract, not a failure. Structural errors (bundle-empty,
+          # path-escape, invalid names, ...) still fail the smoke. A provider
+          # whose every module failed import also reports no components, which
+          # is fine here; the bundle's own tests cover behavior with SDKs.
+          if "lfx.bundles" in eps:
+              from lfx.extension import load_lfx_bundles_extensions
+
+              EXPECTED_WITHOUT_EXTRAS = {"module-import-failed"}
+              results = load_lfx_bundles_extensions()
+              bad = [e.code for r in results for e in r.errors if e.code not in EXPECTED_WITHOUT_EXTRAS]
+              assert not bad, f"{name}: lfx.bundles structural discovery errors: {bad}"
+              providers = sum(1 for r in results if r.bundle)
+              degraded = sum(1 for r in results for e in r.errors if e.code in EXPECTED_WITHOUT_EXTRAS)
+              print(f"  lfx.bundles discovery OK ({providers} providers; {degraded} module(s) degraded sans extras)")
+          print(f"{name}: contract smoke OK")
+          PY
+
+      - name: Validate manifest (manifest-shipping bundles only)
+        run: |
+          # Manifest bundles ship extension.json; the manifest-less metapackage
+          # has none and is exempt from `lfx extension validate`.
+          manifest=$(ls "${{ matrix.bundle }}"/src/*/extension.json 2>/dev/null | head -1 || true)
+          if [ -n "$manifest" ]; then
+            .venv/bin/lfx extension validate "$(dirname "$manifest")"
+          else
+            echo "manifest-less bundle; skipping extension validate"
+          fi
+
+      - name: Run the bundle's own tests
+        run: |
+          if [ -d "${{ matrix.bundle }}/tests" ]; then
+            .venv/bin/python -m pytest "${{ matrix.bundle }}/tests" -q
+          else
+            echo "no tests/ directory in ${{ matrix.bundle }}; skipping"
+          fi

src/bundles/ibm/tests/test_optional_dependency.py

@@ -17,22 +17,38 @@
 
 import pytest
 
-# Bundle modules whose module-level code must re-run under the simulated
-# absence; dropping them from the cache forces a fresh import.
-_BUNDLE_MODULES = (
-    "lfx_ibm.components.ibm",
-    "lfx_ibm.components.ibm.db2vs",
-    "lfx_ibm.components.ibm.db2_vector",
-)
+
+def _bundle_module_names() -> list[str]:
+    return [name for name in sys.modules if name == "lfx_ibm" or name.startswith("lfx_ibm.")]
 
 
 @pytest.fixture
-def without_ibm_db(monkeypatch):
-    """Make the ibm-db driver look uninstalled (as on linux/aarch64)."""
-    monkeypatch.setitem(sys.modules, "ibm_db", None)
-    monkeypatch.setitem(sys.modules, "ibm_db_dbi", None)
-    for name in _BUNDLE_MODULES:
-        monkeypatch.delitem(sys.modules, name, raising=False)
+def without_ibm_db():
+    """Make the ibm-db driver look uninstalled (as on linux/aarch64).
+
+    The whole ``lfx_ibm`` module tree is snapshotted, dropped, and restored
+    wholesale. A partial delete-and-reimport (the previous monkeypatch
+    approach) leaves ``sys.modules`` and the parent packages' submodule
+    attribute bindings incoherent for the *next* test: entries created
+    during the test survive teardown, and a re-imported parent never regains
+    the submodule attributes that ``mock.patch`` target resolution walks on
+    Python 3.10 (3.11+ mock resolves via ``sys.modules`` and tolerates it).
+    """
+    saved = {name: sys.modules[name] for name in _bundle_module_names()}
+    for name in saved:
+        del sys.modules[name]
+    sys.modules["ibm_db"] = None
+    sys.modules["ibm_db_dbi"] = None
+    try:
+        yield
+    finally:
+        for name in ("ibm_db", "ibm_db_dbi"):
+            sys.modules.pop(name, None)
+        # Drop everything imported under the simulated absence, then put the
+        # original, mutually-consistent module tree back.
+        for name in _bundle_module_names():
+            del sys.modules[name]
+        sys.modules.update(saved)
 
 
 @pytest.mark.usefixtures("without_ibm_db")