Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ Each OpenData databases has a high level data flow that looks like this:
Object Storage
```
1. The API layer is specific to each database, and implements the appropriate read and write APIs for the database. When the API layer receives write requests, they are forwarded to the Ingestor component. For instance, a TSDB may implement the OTEL protocol at the API layer which receives OTEL write requests.
2. The Ingestors receive writes coming from the API layer, convert them to Write-Ahead-Log (WAL) records, batch these records and flush them to object storage, and then update the Metadata. The WAL record formats may differ for different database types. We describe the metadata layer in detail later, but for now it suffices to know that the matadata contains locations and versions of all files in the system. The metadata lives in object storage, and all OpenData systems use the same components to enable writing metadata updates atomically. Atomic updates to metadata ensure that readers and writers in the system operate on consistent views of the data in the system.
2. The Ingestors receive writes coming from the API layer, convert them to Write-Ahead-Log (WAL) records, batch these records and flush them to object storage, and then update the Metadata. The WAL record formats may differ for different database types. We describe the metadata layer in detail later, but for now it suffices to know that the metadata contains locations and versions of all files in the system. The metadata lives in object storage, and all OpenData systems use the same components to enable writing metadata updates atomically. Atomic updates to metadata ensure that readers and writers in the system operate on consistent views of the data in the system.
3. There are two types of Compactors in the OpenData system. One type is called the WAL compactor. It takes new WAL files written by ingestors and write the WAL entries into a format that's optimized to serve the queries of each database. For example, a compactor for a TSDB will take a WAL file and write inverted indexes, dictionaries, etc. These read-optimized files are called Collections.
4. The other type of Compactor takes existing Collections and rewrites them into new Collections. These Compactors are used to drop deleted data, to improve data locality, etc. The logic of the Compactor is database specific. The goal of the background compaction process is to maintain read optimized versions of the data across updates.
5. The query executors are responsible for serving queries on the system. These are custom for database. For instance, the query executor of our TSDB implements PromQL. When executing a query, the executors lookup the metadata to find the files relevant to serving the query, and retrieves the data from those files. The main job of the query executor is to keep the right indexes in memory and the hot data cached so that queries on the system are both fast and do not create unduly expensive calls to the objectstore.
Expand Down