Skip to content

feat(ai): add Valkey vector store backend - #1544

Open
atao2004 wants to merge 13 commits into
conductor-oss:mainfrom
atao2004:add-valkey-vector-store
Open

feat(ai): add Valkey vector store backend#1544
atao2004 wants to merge 13 commits into
conductor-oss:mainfrom
atao2004:add-valkey-vector-store

Conversation

@atao2004

Copy link
Copy Markdown

Pull Request type

  • Feature

Summary

Adds Valkey as a VectorDB backend for conductor-ai.

This enables Conductor AI workloads to use a Valkey instance with the valkey-search module for
vector indexing and KNN search through the Valkey GLIDE Java client. It is intended for teams
already operating Valkey who want to use that infrastructure for vector search as well.

Scope is standalone Valkey only. Cluster-mode support is intentionally deferred.

Relates to #1439

Changes made

  • Added ValkeyVectorDB and ValkeyConfig, implementing Conductor's VectorDB abstraction.

    • Uses Valkey Search FT.CREATE and FT.SEARCH KNN commands through GLIDE.
    • Uses FLOAT32 little-endian vector encoding for both document writes and query vectors.
    • Supports configurable dimensions, distance metric (cosine, l2, ip), indexing method,
      key prefix, TLS, authentication, database, and request timeout.
  • Registered valkey and valkeyvectordb as VectorDBInstanceConfig types.

    • This makes Valkey available through Conductor's normal named VectorDB configuration and
      provider lookup path.
  • Added real Valkey Search integration coverage with Testcontainers.

    • Uses valkey/valkey-bundle:9.1.2, which includes the valkey-search module.
    • Covers KNN distance ordering, metadata and parent-ID round trips, upserts, namespace
      isolation, result limits, concurrent first writes, reuse of an existing index, and dimension
      validation.
    • Covers VectorDBInstanceConfig -> VectorDBProvider -> ValkeyVectorDB -> GLIDE -> valkey-search with a focused provider-path integration test.
  • Added configuration and usage documentation for the Valkey backend.

Vector-search evidence

The screenshot test uses a real Valkey Search server started by Testcontainers; it does not mock
Valkey, GLIDE, FT.CREATE, vector writes, or FT.SEARCH.

The focused integration test configures a Valkey instance through Conductor's normal provider path,
writes deterministic embeddings, and performs KNN search:

Document Vector Returned cosine distance
doc-a [1, 0, 0, 0] 0.0
doc-b [0.9, 0.1, 0, 0] 0.006116...
doc-c [0, 1, 0, 0] 1.0

Query vector: [1, 0, 0, 0]

The test verifies the expected nearest-neighbor order:

doc-a -> doc-b -> doc-c

Reproduce

Prerequisites:

  • Java 21+
  • Docker or another Docker-API-compatible container runtime reachable by Testcontainers

Run:

./gradlew :conductor-ai:spotlessApply :conductor-ai:test \
  --tests org.conductoross.conductor.ai.vectordb.ValkeyVectorDBRoundTripTest \
  --info

Expected screenshot-visible output:

VALKEY KNN EVIDENCE | query=[1.0, 0.0, 0.0, 0.0] |
results=[doc-a:0.0, doc-b:0.00611627101898, doc-c:1.0]

BUILD SUCCESSFUL
Screenshot 2026-08-17 at 12 21 23 PM Screenshot 2026-08-17 at 12 24 03 PM

For a persistent Valkey Search instance containing the demo index, inspect the index with:

valkey-cli FT.INFO provider-evidence:docs:ns

Testing

./gradlew :conductor-ai:spotlessApply :conductor-ai:test \
  --tests org.conductoross.conductor.ai.vectordb.ValkeyVectorDBRoundTripTest \
  --info

Passed locally.

Files considered but not changed

  • Other VectorDB backends: no behavior change intended; this PR adds Valkey through the existing
    VectorDB extension point.
  • Cluster-mode client support: deliberately not included. ValkeyVectorDB is standalone-only in
    this PR.
  • Production logging: not changed for screenshots. The evidence output is test-only.

atao2004 added 13 commits August 6, 2026 11:47
- New ValkeyVectorDB provider using valkey-search FT.CREATE/FT.SEARCH via
  io.valkey:valkey-glide:2.5.0
- Physical index derived per (indexName, namespace) so namespaces are isolated
- Scores are raw cosine distance (lower is better), matching pgvector/sqlite-vec
- Embeddings encoded once as little-endian FLOAT32 for both writes and queries
- Provider-agnostic @PreDestroy hook in VectorDBProvider closes Closeable instances
- 78 unit tests covering encoding, validation, lifecycle, and error classification

Signed-off-by: Anna Tao <annatao2004@gmail.com>
- Exercises the provider against a real valkey-search server via
  valkey/valkey-bundle:9.1.2, pinned for reproducibility
- Covers KNN distance ordering, field and metadata round trip, upsert
  replacement, maxResults beyond the default limit of 10, namespace
  isolation under a shared index name, concurrent first writes, and index
  reuse by a second instance
- Host and mapped port are read from the container, so the test runs
  unchanged alongside an existing Valkey on the default port
- Declares org.testcontainers:testcontainers explicitly rather than relying
  on it transitively

Signed-off-by: Anna Tao <annatao2004@gmail.com>
- Add a Valkey section to VECTORDB_CONFIGURATION.md: configuration example,
  all 11 properties with defaults and validation constraints, key and index
  naming scheme, and troubleshooting entries
- Record that score is a raw cosine distance where lower is closer, and warn
  that MongoDB and Pinecone return the opposite convention
- State the standalone-only scope and the valkey-search module requirement
- Source the password from the environment in the example rather than inline
- List Valkey alongside the existing providers across the AI and concept docs,
  updating the vector database count from three to four

Signed-off-by: Anna Tao <annatao2004@gmail.com>
- Fail startup instead of silently dropping a vector DB instance whose
  configuration or construction throws (VectorDBProvider,
  VectorDBInstanceConfig)
- Anchor Valkey already-exists/unknown-command error classification to
  the verified GLIDE message shape instead of free-floating substring
  matches
- Validate document ids against the same allowlist used for
  indexName/namespace before key construction
- Set a GLIDE clientName for CLIENT LIST observability on shared
  Valkey servers

Signed-off-by: Anna Tao <annatao2004@gmail.com>
Test comments referenced internal QA-review finding IDs (M0/M1/N1/L1)
that only make sense with the review doc, which isn't part of this
PR. Rewritten to describe the behavior each test verifies on its own.

Signed-off-by: Anna Tao <annatao2004@gmail.com>
- Close already-created vector DB instances before throwing the
  startup aggregate exception, instead of leaking their connections
- Only fail startup for configuration errors; log and skip runtime/
  connectivity failures, matching how Postgres/Mongo/Pinecone already
  behave (Valkey is the only backend that connects eagerly)
- Clear VectorDBProvider's map after dispose() so a lookup after
  shutdown returns null instead of an already-closed instance
- Preserve the stack trace in the dispose() close-failure log
- Reject a blank/null Valkey instance name with a clear error instead
  of an opaque NullPointerException

Signed-off-by: Anna Tao <annatao2004@gmail.com>
Catching bare RuntimeException to tolerate a transient Valkey connection
failure at startup also silently swallowed genuine bugs (NPE,
ClassCastException) from any backend. Add ValkeyConnectionException as a
distinct type thrown only for GLIDE connection-establishment failures,
and narrow VectorDBInstanceConfig's skip-path catch to that type so any
other RuntimeException still fails startup loudly.

Signed-off-by: Anna Tao <annatao2004@gmail.com>
- GLIDE client creation (TCP+TLS+AUTH+SELECT) routinely takes longer
  than a single command, especially cross-AZ. Give it its own timeout
  (creationTimeoutMs = max(requestTimeoutMs * 3, 5000ms)) instead of
  reusing the per-command requestTimeoutMs, which could spuriously
  trip and cause a healthy instance to be skipped at startup.
- Close a narrow race in VectorDBProvider: a concurrent get() call
  could return an instance that dispose() was in the middle of
  closing. Set a disposed flag at the start of dispose(), checked in
  get(), so lookups start returning null immediately rather than
  racing the close loop.

Signed-off-by: Anna Tao <annatao2004@gmail.com>
Move FT.CREATE network I/O outside ConcurrentHashMap mapping functions.\n\nUse deadline-based polling with diagnostic timeout failures in Valkey round-trip tests.

Signed-off-by: Anna Tao <annatao2004@gmail.com>
- Support GLIDE alternating key-value arrays
- Validate dimensions from existing vector indexes
- Add regression coverage for the runtime response shape

Signed-off-by: Anna Tao <annatao2004@gmail.com>
Signed-off-by: Anna Tao <annatao2004@gmail.com>
Signed-off-by: Anna Tao <annatao2004@gmail.com>
Signed-off-by: Anna Tao <annatao2004@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant