Add readme - #1
Merged
Merged
Conversation
apurvam
commented
Dec 15, 2025
agavra
reviewed
Dec 15, 2025
apurvam
commented
Dec 17, 2025
agavra
approved these changes
Dec 17, 2025
agavra
left a comment
agavra
left a comment
Contributor
There was a problem hiding this comment.
This looks really good to me! I'm happy with this as the first cut for the README.
I do think we should take a pass through and reduce word count since it's a wordy at times. Left some nits inline.
Comment on lines
+85
to
+87
| # License | ||
|
|
||
| MIT License |
Contributor
There was a problem hiding this comment.
we can drop this, we have a LICENSE.md
hachikuji
reviewed
Dec 17, 2025
|
|
||
| 1. The API layer is specific to each database, and implements the appropriate read and write APIs for the database. When the API layer receives write requests, they are forwarded to the Ingestor services. | ||
| 2. The Ingestor services are stateless. They batch writes coming from the API layer, flush the data in a Write-Ahead-Log (WAL) format which is optimized for high throughput writes to object storage, and update the Metadata. We describe the metadata layer in detail later, but for now it suffices to know that the matadata contains locations and versions of all files in the system. The metadata lives in objectstorage, and all OpenData systems use the same components to enable writing metadata updates atomically. Atomic updates to metadata ensure that readers and writers in the system operate on consistent views of the data in the system. The Ingestors are fairly similar across database types. | ||
| 3. There are two types of Compactors in the OpenData system. One type is called the WAL compactor. It takes new WAL files written by ingestors and write the WAL entries into a format that's optimized to serve the queries of each database. For example, a compactor for a TSDB will take a WAL file and write inverted indexes, dictionaries, etc. These read-optimized files are called Collections. |
Contributor
There was a problem hiding this comment.
This feels a little deep for the README. Perhaps a separate architecture document could get into the details of systems on top of SlateDb. I wonder if this doc could focus on the core abstractions: storage as key-values, readers (ingest), writers (query). My mental model of opendata is structured like this:
- Root: The root of the system is SlateDb which implements the LSM abstraction and provides the hooks to control its behavior.
- Core Data Structures: on top of SlateDb, we have the core storage modules. This basically defines the SlateDb record mapping and the core API to consume the data structure. We treat tsdb and log as two core data structures. The API of these data structures is modeled after SlateDb.
- Ingest: On top of core data structures, we define ingest APIs which are basically providing the server abstraction to take client requests and invoke the write APIs on the underlying data structure. OTEL is an ingest layer for tsdb. For the log, we could build a Kafka adapter which accepts
Producerequests. - Query: On top of core data structures, we also expose higher-level query functionality. While the core of tsdb data access may be primitive, we would define a prometheus query layer which works with PromQL. Or for the log, we could define a Kafka layer which works like a consumer or accepts
Fetchrequests.
Contributor
Author
There was a problem hiding this comment.
This makes sense. I added a 10,000ft view of the architecture that is along these lines. I didn't get into the specifics of building custom data structures on top of slateDB, aligning APIs, etc, since that will get too detailed.
hachikuji
approved these changes
Dec 18, 2025
hachikuji
left a comment
hachikuji
left a comment
Contributor
There was a problem hiding this comment.
LGTM. Just one nit. I wonder if we can use the ascii diagrams in the README. They are easier to evolve than pngs and easier to view in editors.
apurvam
added a commit
that referenced
this pull request
Feb 9, 2026
- Cap long-poll sleep to remaining time before deadline so we don't oversleep when timeout < poll_interval (review comment #1) - Add comment explaining why #[serde(default)] is needed on the `follow` field — bool requires it unlike Option<T> (review comment #2) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Venkat2811
referenced
this pull request
in Venkat2811/opendata
May 14, 2026
`VectorDb::query_engine()` previously acquired the same tokio Mutex
on every query at `db.rs:840` to read the snapshot. Every query, no
matter how trivial, serialized through that lock. The snapshot is
read-mostly + written only at flush time, so wrapping it in
`arc_swap::ArcSwap` allows lock-free reads while preserving atomic
publication of new snapshots.
This patch:
- Adds `arc-swap = "1"` to workspace deps and `vector/Cargo.toml`.
Lockfile resolves to 1.9.1.
- Switches the `LastAppliedSnapshot` wrapper from
`Arc<Mutex<LastAppliedSnapshot>>` to
`Arc<ArcSwap<LastAppliedSnapshot>>`.
- Read paths (`query_engine`, `num_centroids`, `validate_cache`)
use `ArcSwap::load()` instead of `lock().expect(...)`.
- Flush path uses `store(Arc::new(...))` instead of `*lock() = ...`.
- `validate_cache` holds the Guard across the await rather than
cloning the snapshot — fewer Arc clones, identical correctness.
- Test fixture in `flusher.rs::tests` mirrors the constructor change.
Drop semantics: per arc-swap's documented behavior, the previous
`Arc<Snapshot>` is dropped only after the last outstanding Guard
referencing it is released. In-flight readers continue to observe
the complete prior snapshot through to completion.
## What changes for callers
Nothing. `LastAppliedSnapshot` is `pub(crate)` and the
`last_applied_snapshot` field is `pub(crate)`. No public-API
surface is affected. This is an internal type swap.
## Why this is correct
1. ArcSwap requires `T: Send + Sync`. `LastAppliedSnapshot` is
`Send + Sync` (its members are all `Arc`s of `Send + Sync`
types), which the compiler verifies.
2. Both the previous Mutex<T> and the new ArcSwap<T> install a
replacement atomically. Either a reader sees V_n or V_{n+1};
never a torn snapshot, in either implementation.
3. The codebase has a single writer per VectorDb: WriteCoordinator
owns one Flusher, so write-write races are not possible. ArcSwap
would serialize them via last-writer-wins regardless.
4. The existing 329 lib tests pass unchanged. Two new
concurrency tests added (below).
## New tests
In `vector/src/db.rs` under `#[cfg(test)] mod tests`, using the
existing `create_test_config()` fixture and a multi-threaded
tokio runtime:
- `snapshot_supports_concurrent_readers`: 16 reader tasks each
issue 25 search queries against a 50-vector index. Asserts
every query returns hits and no task panics. Surfaces
deadlocks, mutex poisoning, or torn-snapshot reads.
- `snapshot_readers_dont_block_on_concurrent_flushes`: 8 reader
tasks hot-loop search queries while 4 write+flush batches
interleave. Asserts each reader completes at least one search
and every result set is non-empty. Exercises the
`ArcSwap::store` publication path under concurrent reads.
Suite runs in 0.14s. 331 lib tests pass (329 prior + 2 new).
## Measured impact
Bench: HTTP wire path against a locally running MinIO-backed
server. cohere1M dataset (1M f32 768-dim cosine vectors), 1000
unique queries from `queries.f32`, k=10, nprobe=100, 100-query
prewarm before timing, 4 concurrency cells. Pre-change run is a
single measurement; post-change ran twice to characterize
run-to-run variance.
Pre-change baseline (single run, Mutex<LastAppliedSnapshot>):
c QPS p99 ms
1 10.17 118
16 9.82 3,308
32 9.72 8,988
64 9.66 26,201
Post-change run #1 (ArcSwap<LastAppliedSnapshot>):
c QPS p99 ms
1 15.33 78
16 15.95 1,509
32 15.76 4,803
64 15.63 16,262
Post-change run opendata-oss#2 (independent run on same setup):
c QPS p99 ms
1 14.98 82
16 15.92 1,507
32 15.43 3,288
64 13.76 8,139
Run-to-run variance: QPS within ~10% across cells; p99 within
3% at c=1 and c=16, but 30-50% at c=32 and c=64 (high-tail
behavior is workload+scheduler-sensitive).
Across both post-change runs:
QPS gain over baseline:
c=1 1.47x - 1.51x
c=16 1.62x - 1.62x
c=32 1.59x - 1.62x
c=64 1.42x - 1.62x
p99 reduction over baseline:
c=1 1.44x - 1.51x (118 ms -> 78-82 ms)
c=16 2.19x - 2.20x (3308 ms -> 1507-1509 ms)
c=32 1.87x - 2.73x (8988 ms -> 3288-4803 ms)
c=64 1.61x - 3.22x (26201 ms -> 8139-16262 ms)
Recall@10: 0.9238 (pre) -> 0.9256 (post, both runs). Unchanged
within noise. Error rate: 0 across all post-change cells; 3
errors at c=64 in the pre-change run.
The patch removes the snapshot lock as a serializer. The QPS
curve remains essentially flat across c={1, 16, 32, 64} after
the change (still rises only marginally with concurrency),
which indicates another serializing element on the read path
upstream of where this patch operates. Identifying it is out
of scope for this change.
## Bench reproducer
Environment used for the numbers above:
- Linux 6.8 x86_64
- AMD Ryzen 7 5800X (8 cores / 16 SMT threads)
- 62 GiB RAM
- NVMe SSD (locally attached)
- rustc 1.97.0-nightly (the `timeseries` workspace member
requires nightly `duration_constructors_lite`)
- MinIO single-node, 127.0.0.1:9000, default settings, fresh
bucket per measurement
Server config (the bench .yaml):
storage:
type: SlateDb
path: vector-data-cohere1m
object_store: { type: Aws, region: us-east-1, bucket: <fresh> }
dimensions: 768
distance_metric: Cosine
flush_interval: 5
split_threshold_vectors: 200
merge_threshold_vectors: 50
split_search_neighbourhood: 16
block_cache_bytes: "1073741824" # 1 GiB
Ingest: HTTP POST /api/v1/vector/write, batch=100 vectors per
request. axum's default 2 MB body limit caps batch size at
~100 768-dim f32 vectors encoded as JSON. 1M vectors =
10000 batches. Ingest wall ≈ 1080 s (~925 vec/s) both pre and
post; the patch does not affect the write path.
Query client: Python stdlib urllib + ThreadPoolExecutor with
N workers. urllib does not pool HTTP connections, so each
query opens a fresh TCP connection. This adds a small
TCP-handshake cost to every query identically across pre and
post measurements; it does not bias the comparison but is
worth noting for absolute p99 numbers.
Query workload: 1000 unique query vectors from the dataset's
queries.f32, k=10, nprobe=100, 100-query prewarm before the
timed loop. Workers split queries round-robin
(`indices[w::workers]`). Each cell ran for ≈ 60-100 seconds
elapsed in the timed phase. Recall computed as mean over
queries of |returned_top_10 ∩ ground_truth_top_10| / 10.
Per-cell numbers reported above are taken from two independent
post-change runs and one pre-change run; baseline run was
single because the codebase before this patch is the published
v0.2.1 tip.
Venkat2811
referenced
this pull request
in Venkat2811/opendata
May 14, 2026
`VectorDb::query_engine()` previously acquired the same tokio Mutex
on every query at `db.rs:840` to read the snapshot. Every query, no
matter how trivial, serialized through that lock. The snapshot is
read-mostly + written only at flush time, so wrapping it in
`arc_swap::ArcSwap` allows lock-free reads while preserving atomic
publication of new snapshots.
This patch:
- Adds `arc-swap = "1"` to workspace deps and `vector/Cargo.toml`.
Lockfile resolves to 1.9.1.
- Switches the `LastAppliedSnapshot` wrapper from
`Arc<Mutex<LastAppliedSnapshot>>` to
`Arc<ArcSwap<LastAppliedSnapshot>>`.
- Read paths (`query_engine`, `num_centroids`, `validate_cache`)
use `ArcSwap::load()` instead of `lock().expect(...)`.
- Flush path uses `store(Arc::new(...))` instead of `*lock() = ...`.
- `validate_cache` holds the Guard across the await rather than
cloning the snapshot — fewer Arc clones, identical correctness.
- Test fixture in `flusher.rs::tests` mirrors the constructor change.
Drop semantics: per arc-swap's documented behavior, the previous
`Arc<Snapshot>` is dropped only after the last outstanding Guard
referencing it is released. In-flight readers continue to observe
the complete prior snapshot through to completion.
## What changes for callers
Nothing. `LastAppliedSnapshot` is `pub(crate)` and the
`last_applied_snapshot` field is `pub(crate)`. No public-API
surface is affected. This is an internal type swap.
## Why this is correct
1. ArcSwap requires `T: Send + Sync`. `LastAppliedSnapshot` is
`Send + Sync` (its members are all `Arc`s of `Send + Sync`
types), which the compiler verifies.
2. Both the previous Mutex<T> and the new ArcSwap<T> install a
replacement atomically. Either a reader sees V_n or V_{n+1};
never a torn snapshot, in either implementation.
3. The codebase has a single writer per VectorDb: WriteCoordinator
owns one Flusher, so write-write races are not possible. ArcSwap
would serialize them via last-writer-wins regardless.
4. The existing 329 lib tests pass unchanged. Two new
concurrency tests added (below).
## New tests
In `vector/src/db.rs` under `#[cfg(test)] mod tests`, using the
existing `create_test_config()` fixture and a multi-threaded
tokio runtime:
- `snapshot_supports_concurrent_readers`: 16 reader tasks each
issue 25 search queries against a 50-vector index. Asserts
every query returns hits and no task panics. Surfaces
deadlocks, mutex poisoning, or torn-snapshot reads.
- `snapshot_readers_dont_block_on_concurrent_flushes`: 8 reader
tasks hot-loop search queries while 4 write+flush batches
interleave. Asserts each reader completes at least one search
and every result set is non-empty. Exercises the
`ArcSwap::store` publication path under concurrent reads.
Suite runs in 0.14s. 331 lib tests pass (329 prior + 2 new).
## Measured impact
Bench: HTTP wire path against a locally running MinIO-backed
server. cohere1M dataset (1M f32 768-dim cosine vectors), 1000
unique queries from `queries.f32`, k=10, nprobe=100, 100-query
prewarm before timing, 4 concurrency cells. Pre-change run is a
single measurement; post-change ran twice to characterize
run-to-run variance.
Pre-change baseline (single run, Mutex<LastAppliedSnapshot>):
c QPS p99 ms
1 10.17 118
16 9.82 3,308
32 9.72 8,988
64 9.66 26,201
Post-change run #1 (ArcSwap<LastAppliedSnapshot>):
c QPS p99 ms
1 15.33 78
16 15.95 1,509
32 15.76 4,803
64 15.63 16,262
Post-change run opendata-oss#2 (independent run on same setup):
c QPS p99 ms
1 14.98 82
16 15.92 1,507
32 15.43 3,288
64 13.76 8,139
Run-to-run variance: QPS within ~10% across cells; p99 within
3% at c=1 and c=16, but 30-50% at c=32 and c=64 (high-tail
behavior is workload+scheduler-sensitive).
Across both post-change runs:
QPS gain over baseline:
c=1 1.47x - 1.51x
c=16 1.62x - 1.62x
c=32 1.59x - 1.62x
c=64 1.42x - 1.62x
p99 reduction over baseline:
c=1 1.44x - 1.51x (118 ms -> 78-82 ms)
c=16 2.19x - 2.20x (3308 ms -> 1507-1509 ms)
c=32 1.87x - 2.73x (8988 ms -> 3288-4803 ms)
c=64 1.61x - 3.22x (26201 ms -> 8139-16262 ms)
Recall@10: 0.9238 (pre) -> 0.9256 (post, both runs). Unchanged
within noise. Error rate: 0 across all post-change cells; 3
errors at c=64 in the pre-change run.
The patch removes the snapshot lock as a serializer. The QPS
curve remains essentially flat across c={1, 16, 32, 64} after
the change (still rises only marginally with concurrency),
which indicates another serializing element on the read path
upstream of where this patch operates. Identifying it is out
of scope for this change.
## Bench reproducer
Environment used for the numbers above:
- Linux 6.8 x86_64
- AMD Ryzen 7 5800X (8 cores / 16 SMT threads)
- 62 GiB RAM
- NVMe SSD (locally attached)
- rustc 1.97.0-nightly (the `timeseries` workspace member
requires nightly `duration_constructors_lite`)
- MinIO single-node, 127.0.0.1:9000, default settings, fresh
bucket per measurement
Server config (the bench .yaml):
storage:
type: SlateDb
path: vector-data-cohere1m
object_store: { type: Aws, region: us-east-1, bucket: <fresh> }
dimensions: 768
distance_metric: Cosine
flush_interval: 5
split_threshold_vectors: 200
merge_threshold_vectors: 50
split_search_neighbourhood: 16
block_cache_bytes: "1073741824" # 1 GiB
Ingest: HTTP POST /api/v1/vector/write, batch=100 vectors per
request. axum's default 2 MB body limit caps batch size at
~100 768-dim f32 vectors encoded as JSON. 1M vectors =
10000 batches. Ingest wall ≈ 1080 s (~925 vec/s) both pre and
post; the patch does not affect the write path.
Query client: Python stdlib urllib + ThreadPoolExecutor with
N workers. urllib does not pool HTTP connections, so each
query opens a fresh TCP connection. This adds a small
TCP-handshake cost to every query identically across pre and
post measurements; it does not bias the comparison but is
worth noting for absolute p99 numbers.
Query workload: 1000 unique query vectors from the dataset's
queries.f32, k=10, nprobe=100, 100-query prewarm before the
timed loop. Workers split queries round-robin
(`indices[w::workers]`). Each cell ran for ≈ 60-100 seconds
elapsed in the timed phase. Recall computed as mean over
queries of |returned_top_10 ∩ ground_truth_top_10| / 10.
Per-cell numbers reported above are taken from two independent
post-change runs and one pre-change run; baseline run was
single because the codebase before this patch is the published
v0.2.1 tip.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.