feat(ai): add Valkey vector store backend - #1544
Open
atao2004 wants to merge 13 commits into
Open
Conversation
- New ValkeyVectorDB provider using valkey-search FT.CREATE/FT.SEARCH via io.valkey:valkey-glide:2.5.0 - Physical index derived per (indexName, namespace) so namespaces are isolated - Scores are raw cosine distance (lower is better), matching pgvector/sqlite-vec - Embeddings encoded once as little-endian FLOAT32 for both writes and queries - Provider-agnostic @PreDestroy hook in VectorDBProvider closes Closeable instances - 78 unit tests covering encoding, validation, lifecycle, and error classification Signed-off-by: Anna Tao <annatao2004@gmail.com>
- Exercises the provider against a real valkey-search server via valkey/valkey-bundle:9.1.2, pinned for reproducibility - Covers KNN distance ordering, field and metadata round trip, upsert replacement, maxResults beyond the default limit of 10, namespace isolation under a shared index name, concurrent first writes, and index reuse by a second instance - Host and mapped port are read from the container, so the test runs unchanged alongside an existing Valkey on the default port - Declares org.testcontainers:testcontainers explicitly rather than relying on it transitively Signed-off-by: Anna Tao <annatao2004@gmail.com>
- Add a Valkey section to VECTORDB_CONFIGURATION.md: configuration example, all 11 properties with defaults and validation constraints, key and index naming scheme, and troubleshooting entries - Record that score is a raw cosine distance where lower is closer, and warn that MongoDB and Pinecone return the opposite convention - State the standalone-only scope and the valkey-search module requirement - Source the password from the environment in the example rather than inline - List Valkey alongside the existing providers across the AI and concept docs, updating the vector database count from three to four Signed-off-by: Anna Tao <annatao2004@gmail.com>
- Fail startup instead of silently dropping a vector DB instance whose configuration or construction throws (VectorDBProvider, VectorDBInstanceConfig) - Anchor Valkey already-exists/unknown-command error classification to the verified GLIDE message shape instead of free-floating substring matches - Validate document ids against the same allowlist used for indexName/namespace before key construction - Set a GLIDE clientName for CLIENT LIST observability on shared Valkey servers Signed-off-by: Anna Tao <annatao2004@gmail.com>
Test comments referenced internal QA-review finding IDs (M0/M1/N1/L1) that only make sense with the review doc, which isn't part of this PR. Rewritten to describe the behavior each test verifies on its own. Signed-off-by: Anna Tao <annatao2004@gmail.com>
- Close already-created vector DB instances before throwing the startup aggregate exception, instead of leaking their connections - Only fail startup for configuration errors; log and skip runtime/ connectivity failures, matching how Postgres/Mongo/Pinecone already behave (Valkey is the only backend that connects eagerly) - Clear VectorDBProvider's map after dispose() so a lookup after shutdown returns null instead of an already-closed instance - Preserve the stack trace in the dispose() close-failure log - Reject a blank/null Valkey instance name with a clear error instead of an opaque NullPointerException Signed-off-by: Anna Tao <annatao2004@gmail.com>
Catching bare RuntimeException to tolerate a transient Valkey connection failure at startup also silently swallowed genuine bugs (NPE, ClassCastException) from any backend. Add ValkeyConnectionException as a distinct type thrown only for GLIDE connection-establishment failures, and narrow VectorDBInstanceConfig's skip-path catch to that type so any other RuntimeException still fails startup loudly. Signed-off-by: Anna Tao <annatao2004@gmail.com>
- GLIDE client creation (TCP+TLS+AUTH+SELECT) routinely takes longer than a single command, especially cross-AZ. Give it its own timeout (creationTimeoutMs = max(requestTimeoutMs * 3, 5000ms)) instead of reusing the per-command requestTimeoutMs, which could spuriously trip and cause a healthy instance to be skipped at startup. - Close a narrow race in VectorDBProvider: a concurrent get() call could return an instance that dispose() was in the middle of closing. Set a disposed flag at the start of dispose(), checked in get(), so lookups start returning null immediately rather than racing the close loop. Signed-off-by: Anna Tao <annatao2004@gmail.com>
Move FT.CREATE network I/O outside ConcurrentHashMap mapping functions.\n\nUse deadline-based polling with diagnostic timeout failures in Valkey round-trip tests. Signed-off-by: Anna Tao <annatao2004@gmail.com>
- Support GLIDE alternating key-value arrays - Validate dimensions from existing vector indexes - Add regression coverage for the runtime response shape Signed-off-by: Anna Tao <annatao2004@gmail.com>
Signed-off-by: Anna Tao <annatao2004@gmail.com>
Signed-off-by: Anna Tao <annatao2004@gmail.com>
Signed-off-by: Anna Tao <annatao2004@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request type
Summary
Adds Valkey as a VectorDB backend for
conductor-ai.This enables Conductor AI workloads to use a Valkey instance with the
valkey-searchmodule forvector indexing and KNN search through the Valkey GLIDE Java client. It is intended for teams
already operating Valkey who want to use that infrastructure for vector search as well.
Scope is standalone Valkey only. Cluster-mode support is intentionally deferred.
Relates to #1439
Changes made
Added
ValkeyVectorDBandValkeyConfig, implementing Conductor'sVectorDBabstraction.FT.CREATEandFT.SEARCHKNN commands through GLIDE.cosine,l2,ip), indexing method,key prefix, TLS, authentication, database, and request timeout.
Registered
valkeyandvalkeyvectordbasVectorDBInstanceConfigtypes.provider lookup path.
Added real Valkey Search integration coverage with Testcontainers.
valkey/valkey-bundle:9.1.2, which includes thevalkey-searchmodule.isolation, result limits, concurrent first writes, reuse of an existing index, and dimension
validation.
VectorDBInstanceConfig -> VectorDBProvider -> ValkeyVectorDB -> GLIDE -> valkey-searchwith a focused provider-path integration test.Added configuration and usage documentation for the Valkey backend.
Vector-search evidence
The screenshot test uses a real Valkey Search server started by Testcontainers; it does not mock
Valkey, GLIDE,
FT.CREATE, vector writes, orFT.SEARCH.The focused integration test configures a Valkey instance through Conductor's normal provider path,
writes deterministic embeddings, and performs KNN search:
doc-a[1, 0, 0, 0]0.0doc-b[0.9, 0.1, 0, 0]0.006116...doc-c[0, 1, 0, 0]1.0Query vector:
[1, 0, 0, 0]The test verifies the expected nearest-neighbor order:
Reproduce
Prerequisites:
Run:
Expected screenshot-visible output:
For a persistent Valkey Search instance containing the demo index, inspect the index with:
Testing
Passed locally.
Files considered but not changed
VectorDBextension point.ValkeyVectorDBis standalone-only inthis PR.