fix(distribution)!: make inline::milvus conditional via ENABLE_INLINE_MILVUS env var (#318)

[RHOAIENG-3916](https://issues.redhat.com/browse/RHOAIENG-3916)

### Description

Must merge after #316.

Makes `inline::milvus` conditional on `ENABLE_INLINE_MILVUS`, off by
default. Users who need it can opt in by setting the env var.
`inline::milvus` is incompatible with multi-worker deployments.

This is a breaking change: `inline::milvus` was previously enabled by
default.

Adds `notes` support to `gen_distro_docs.py` so provider-level warnings
appear in the auto-generated README table.

### How Has This Been Tested?

```bash
./tests/smoke.sh
./tests/test_providers.sh
```

### Test Impact

New `test_providers.sh` and CI workflow step. Separate from `smoke.sh`
because provider tests need to start containers with different env var
inputs without affecting the smoke test environment.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Milvus vector provider (`inline::milvus`) is now disabled by default;
enable via the ENABLE_INLINE_MILVUS environment variable. Docs now note
it is incompatible with multi-worker deployments.

* **Tests**
* Added an integration test that verifies provider registration with and
without ENABLE_INLINE_MILVUS.

* **Chores**
* CI: added a "Provider tests" step to run the provider validation
script during build/test.

* **Documentation tooling**
* Release notes generator updated to surface provider notes in the docs.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Author: derekhiggins

.github/workflows/redhat-distro-container.yml

@@ -200,6 +200,14 @@ jobs:
         shell: bash
         run: ./tests/smoke.sh
 
+      - name: Provider tests
+        if: github.event_name != 'workflow_dispatch'
+        id: provider-tests
+        shell: bash
+        env:
+          IMAGE_TAG: ${{ contains(fromJSON('["workflow_dispatch", "schedule"]'), github.event_name) && format('source-{0}-{1}', env.LLAMA_STACK_COMMIT_SHA, github.sha) || github.sha }}
+        run: ./tests/test_providers.sh
+
       - name: Integration tests
         if: github.event_name != 'workflow_dispatch' && (matrix.arch == 'amd64' || env.USING_MAAS == 'true')
         id: integration-tests

distribution/README.md

@@ -38,7 +38,7 @@ You can see an overview of the APIs and Providers the image ships with in the ta
 | tool_runtime | remote::model-context-protocol | No | ✅ | N/A |
 | tool_runtime | remote::tavily-search | No | ✅ | N/A |
 | vector_io | inline::faiss | No | ❌ | Set the `ENABLE_FAISS` environment variable |
-| vector_io | inline::milvus | No | ✅ | N/A |
+| vector_io | inline::milvus | No | ❌ | Set the `ENABLE_INLINE_MILVUS` environment variable. Incompatible with multi-worker deployments |
 | vector_io | remote::milvus | No | ❌ | Set the `MILVUS_ENDPOINT` environment variable |
 | vector_io | remote::pgvector | No | ❌ | Set the `ENABLE_PGVECTOR` environment variable |
 | vector_io | remote::qdrant | No | ❌ | Set the `ENABLE_QDRANT` environment variable |

distribution/config.yaml

@@ -58,8 +58,9 @@ providers:
       api_key: ${env.OPENAI_API_KEY:=}
       base_url: ${env.OPENAI_BASE_URL:=https://api.openai.com/v1}
   vector_io:
-  - provider_id: milvus
+  - provider_id: ${env.ENABLE_INLINE_MILVUS:+milvus}
     provider_type: inline::milvus
+    notes: Incompatible with multi-worker deployments
     config:
       db_path: /opt/app-root/src/.llama/distributions/rh/milvus.db
       persistence:

scripts/gen_distro_docs.py

@@ -136,6 +136,10 @@ def gen_distro_table(providers_data):
                         enabled_by_default = "✅"
                         how_to_enable = "N/A"
 
+                    notes = provider.get("notes", "")
+                    if notes:
+                        how_to_enable += f". {notes}"
+
                     # Determine external status using config.yaml data
                     external_status = external_providers.get(provider_type, "No")
 

tests/test_providers.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+set -uo pipefail
+
+# Test that conditional provider loading respects env vars.
+# Each test starts its own container with its own name and port.
+
+function check_provider {
+  local port="$1" provider_type="$2" expect="$3" name="$4"
+  resp=$(curl -fsS "http://127.0.0.1:${port}/v1/providers" 2>/dev/null) || true
+  match=$(echo "$resp" | jq --arg pt "$provider_type" \
+    '[.data[] | select(.provider_type == $pt)] | length')
+  match=${match:-0}
+  if [ "$expect" = "present" ] && [ "$match" -gt 0 ]; then return 0; fi
+  if [ "$expect" = "absent" ] && [ "$match" -eq 0 ]; then return 0; fi
+  echo "FAIL: $provider_type expected $expect"; docker logs "$name" || true; return 1
+}
+
+function run_container {
+  local name="$1" port="$2"; shift 2
+  docker rm -f "$name" 2>/dev/null || true
+  docker run -d --pull=never --net=host --name "$name" "$@" \
+    "$IMAGE_NAME:${IMAGE_TAG:-$GITHUB_SHA}" --port "$port"
+  for _ in {1..60}; do
+    curl -fsS "http://127.0.0.1:${port}/v1/health" 2>/dev/null && return 0
+    sleep 1
+  done
+  echo "Server $name failed to start"; docker logs "$name" || true; return 1
+}
+
+failed=()
+
+run_container pt-no-milvus 8421 && \
+  check_provider 8421 inline::milvus absent pt-no-milvus || failed+=(milvus-absent)
+docker rm -f pt-no-milvus 2>/dev/null || true
+
+run_container pt-with-milvus 8422 --env ENABLE_INLINE_MILVUS=true && \
+  check_provider 8422 inline::milvus present pt-with-milvus || failed+=(milvus-present)
+docker rm -f pt-with-milvus 2>/dev/null || true
+
+if [ ${#failed[@]} -eq 0 ]; then echo "Provider tests passed"; exit 0; fi
+echo "Provider tests failed: ${failed[*]}"; exit 1