feat: Full support for Astra DB as a backend

[erichare]

This pull request introduces support for Astra DB as an alternative vector backend to OpenSearch, allowing dynamic selection of the knowledge backend and associated flows. The changes include configuration updates across Docker, Helm charts, and Python code to enable backend selection, Astra DB credentials, and flow definitions. The API is updated to expose the selected backend and to use the appropriate flow IDs and files based on the backend.

Key changes include:

Backend Selection and Configuration:

• Added environment variables and Helm chart values for VECTOR_BACKEND, ASTRA_DB_APPLICATION_TOKEN, and ASTRA_DB_API_ENDPOINT to support Astra DB as a vector backend, in addition to OpenSearch. (docker-compose.yml [1] [2] [3]; kubernetes/helm/openrag/values.yaml [4] [5]; kubernetes/helm/openrag/templates/backend/backend-dotenv.yaml [6]; kubernetes/helm/openrag/templates/langflow/langflow-dotenv.yaml [7]
• Updated config and environment variable loading logic to recognize and propagate the backend selection and Astra DB credentials throughout the system. (src/config/config_manager.py [1] [2]; src/config/settings.py [3] [4]

Flow Definitions and Dynamic Flow Selection:

• Introduced FlowDefinition dataclass and dictionaries for OpenSearch and Astra DB flow definitions, including flow IDs, filenames, and vector store node IDs. This enables dynamic flow selection based on the active backend. (src/config/settings.py src/config/settings.pyR49-R113)
• Updated logic in the API to use helper functions for retrieving the active flow IDs and backend, ensuring the correct flows are used and exposed in API responses. (src/api/settings.py [1] [2] [3] [4]

API and Model Updates:

• Exposed the selected knowledge backend in API responses and models, ensuring clients are aware of which backend is active. (src/api/settings.py [1] [2]; src/api/v1/settings.py [3] [4] [5]
• Modified the document existence check endpoint to short-circuit and always return "not exists" if Astra DB is the backend (since filename-based existence checks are not supported for Astra). (src/api/documents.py src/api/documents.pyL120-R127)

Helm and Flow File Updates:

• Added Astra-specific flow files to Helm chart values and ensured Astra DB variables are included in the set of environment variables exposed to flows. (kubernetes/helm/openrag/values.yaml [1] [2]; kubernetes/helm/openrag/templates/configmaps/app-config-configmap.yaml [3]

These changes collectively enable the system to support both OpenSearch and Astra DB as vector backends, with dynamic configuration and flow selection based on the chosen backend.

[github-actions]

Build successful! ✅
Deploying docs draft.
Deploy successful! View draft