Skip to content

fix: Remove get_cached() and use get() everywhere for cross-worker visibility - #6388

Draft
mattf wants to merge 1 commit into
ogx-ai:mainfrom
mattf:fix-5008
Draft

fix: Remove get_cached() and use get() everywhere for cross-worker visibility#6388
mattf wants to merge 1 commit into
ogx-ai:mainfrom
mattf:fix-5008

Conversation

@mattf

@mattf mattf commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

Remove get_cached() entirely from the DistributionRegistry protocol and implementations. All callers now use get() which implements cache-then-DB fallback semantics.

The old get_cached() was a cache-only accessor that returned None when a routing table lookup missed the in-memory dict. In single-process deployments this was fine, but in multi-worker uvicorn deployments each process has its own cache that is only updated by _ensure_initialized() at startup, register(), update(), and delete() -- all within a single process. A cache miss in worker A after worker B created a resource would always return None instead of falling back to the DB where worker B had persisted the object.

This is the exact bug the register() override already worked around: its doc comment explicitly says 'use super().get() (DB read) rather than self.get_cached() so that in multi-worker deployments, where each process has its own in-memory cache, we always read the authoritative stored object'. The same logic applies to all callers.

get_all() already had TTL-based cache refresh on every call, and that's used for list endpoints. But the routing path (get_provider_impl()) and the metrics helper (_get_provider_id()) both called get_cached(), creating a silent failure mode where freshly created resources were invisible until the next TTL refresh or a restart.

By removing get_cached() and using get() everywhere (which checks cache first, falls back to DB on miss, and updates the cache), we get immediate cross-worker visibility for all operations with only a modest additional DB round-trip cost on first access per resource (and that access gets cached for subsequent calls).

Changes:

  • Remove get_cached() from DistributionRegistry protocol
  • Remove get_cached() from DiskDistributionRegistry (was raising NotImplementedError)
  • Remove get_cached() from CachedDiskDistributionRegistry (was just a dict get)
  • Update get_provider_impl() in common.py to use get() (was using get_cached())
  • Update _get_provider_id() in vector_io.py to be async and use get() (was sync and using get_cached()) -- all 6 call sites are in async methods and already awaited
  • Update test mocks and test assertions that referenced get_cached()

closes #5008

…sibility

Remove get_cached() entirely from the DistributionRegistry protocol and
implementations. All callers now use get() which implements cache-then-DB
fallback semantics.

The old get_cached() was a cache-only accessor that returned None when a
routing table lookup missed the in-memory dict. In single-process
deployments this was fine, but in multi-worker uvicorn deployments each
process has its own cache that is only updated by _ensure_initialized()
at startup, register(), update(), and delete() -- all within a single
process. A cache miss in worker A after worker B created a resource would
always return None instead of falling back to the DB where worker B had
persisted the object.

This is the exact bug the register() override already worked around: its
doc comment explicitly says 'use super().get() (DB read) rather than
self.get_cached() so that in multi-worker deployments, where each process
has its own in-memory cache, we always read the authoritative stored
object'. The same logic applies to all callers.

get_all() already had TTL-based cache refresh on every call, and that's
used for list endpoints. But the routing path (get_provider_impl()) and
the metrics helper (_get_provider_id()) both called get_cached(),
creating a silent failure mode where freshly created resources were
invisible until the next TTL refresh or a restart.

By removing get_cached() and using get() everywhere (which checks cache
first, falls back to DB on miss, and updates the cache), we get
immediate cross-worker visibility for all operations with only a modest
additional DB round-trip cost on first access per resource (and that
access gets cached for subsequent calls).

Changes:
- Remove get_cached() from DistributionRegistry protocol
- Remove get_cached() from DiskDistributionRegistry (was raising
  NotImplementedError)
- Remove get_cached() from CachedDiskDistributionRegistry (was just a
  dict get)
- Update get_provider_impl() in common.py to use get() (was using
  get_cached())
- Update _get_provider_id() in vector_io.py to be async and use get()
  (was sync and using get_cached()) -- all 6 call sites are in async
  methods and already awaited
- Update test mocks and test assertions that referenced get_cached()

Signed-off-by: Matthew Farrellee <matt@cs.wisc.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Vector stores return "not found" immediately after creation when using multiple server workers

1 participant