Add RFC for low-level timseries write API - #16
Conversation
agavra
left a comment
agavra
left a comment
There was a problem hiding this comment.
This looks great, thanks @hachikuji! No serious comments or concerns. Only question around await_durable and whether that's necessary for timeseries ingestion.
| pub struct WriteOptions { | ||
| /// Wait for data to be flushed to durable storage before returning. | ||
| /// Default: false (return immediately after buffering) | ||
| pub await_durable: bool, |
There was a problem hiding this comment.
I'm curious if we need this for timeseries data - right now it's not super straightforward to get a handler to await durable with the batching mechanism I implemented. It would complicate the code with some annoying async book-keeping.
There was a problem hiding this comment.
I didn't see the harm exposing it at the lower level. I think our default would just be false.
There was a problem hiding this comment.
Perhaps some users would want to write billing telemetry at this level and might like a strong guarantee.
There was a problem hiding this comment.
That's fair - I think we can eventually support it. Maybe we'll wait to implement it until we have slatedb-wal factored out or a pluggable slate memtable so that we don't need to handle the synchronization between the buffer and slate's await_durable mechanism ourselves (it was a bit of a pain in slatedb to handle that and i really don't want to custom implement it if we can avoid it).
apurvam
left a comment
apurvam
left a comment
There was a problem hiding this comment.
LGTM! I like the principles laid out, and the RFC exemplifies them well.
3 participants
This RFC proposes a low-level
TimeSeriesAPI which is analogous to SlateDb'sDband OpenData'sLog. The idea is to have a simple, permissive low-level API with all the complexity from systems like OTEL and Prometheus layered on top. I started with the write API because that is relatively simple compared to the read side.I am hoping that each of our systems can expose similar low-level API with conventions and semantics which are roughly inline with SlateDb itself. Basically it should look and feel like SlateDb, but with a custom data model. We can then add APIs which cross-cut each system in higher-level traits. For example, we would probably introduce some trait to expose SlateDb's underlying checkpoint system.