Log sequence tracking RFC

[hachikuji]

The initial log RFC is vague about sequence number tracking. This patch suggests a block-based approach implemented on top of SlateDb.

[agavra]

I'm wondering why we don't use the sequence number maintained by Slate itself? I believe we want to expose that anyway (see slatedb/slatedb#1138 (comment) - it's pretty deep in there but the proposal is good)

[agavra]

ah! just saw your comment at the bottom. yeah that makes sense to me as an interim thing, we might want to just prioritize the slate ticket thought

[agavra]

LGTM, though I think we should consider prioritizing the SlateDB change to expose the internal seqnum

[hachikuji]

@agavra To be honest, I'm on the fence about exposing the sequence number in SlateDb. I guess my concern is how it ties down the implementation for the future. Perhaps as long as it works within a multi-writer context, it's fine. And maybe we have to do it anyway for some use cases. I'll read the thread and try to get a better read on it.

[apurvam]

LGTM! I agree with almog's comment about prioritizing generalizing and reusing the slateDB sequence number if it is already playing a similar role. That would be in the spirit of the substrate.

[criccomini]

LGTM, though I think we should consider prioritizing the SlateDB change to expose the internal seqnum

I have been wanting to do this for a long time but never got around to it. If you guys want to PR it, that'd be 🔥 .

[hachikuji]

I looked at the suggestion in the comment here: slatedb/slatedb#1138 (comment). The API looks reasonable to me (I'll see if I can pick it up since I've been looking for some slatedb work), but it wouldn't solve our problem here. We need the sequence number to be embedded in the key, which means we need to know what the sequence number will be before writing. Getting keys to align with writes seems a little slippery to me. There is also a benefit to decoupling the two. The nice thing about the LSM representation is that it would be possible to overwrite a bad record. That wouldn't be possible anymore if we were tied to the internal sequence number.

[agavra]

We need the sequence number to be embedded in the key, which means we need to know what the sequence number will be before writing.

🤦 yes... that makes sense to me. Though FWIW this could be a use case for decoupling the WAL and WAL compactors in Rohan's terminology. You know the seqnum when its inserted into the WAL, we could then transform that into a full key when it's written to L0 using the sequence number.

Anyway, I'm OK decoupling the two. I imagine it will make things easier in the long run.