RFC for log segmentation by hachikuji · Pull Request #32 · opendata-oss/opendata

hachikuji · 2026-01-07T23:50:44Z

This patch adds a logical segmentation notion which is similar to the way that time buckets work in timeseries. Segments can be created based on time-based boundaries, such as hourly like with timeseries. The user can also seal a segment based on their own application logic. This provides a useful foundation for cross-key operations. For example, when reading from a range of keys, we can use the segment to position reads across all keys in the range. It could also be used as a basis for performing cross-key retention semantics (e.g. drop all segments older than one day).

agavra

Mostly LGTM, some food for thought before the green check though. Biggest concern is adding 8 bytes to each key.

agavra · 2026-01-08T00:00:21Z

+
+```
+Log Entry:
+  | version (u8) | type (u8) | segment_id (u64 BE) | key (TerminatedBytes) | sequence (u64 BE) |


do we need a full u64 for segment id? u32 would allow 4B segments. adding this to every key seems pretty significant for a log structure. There are even varint encodings that maintain lexicographical ordering (see https://github.com/khonsulabs/ordered-varint for example, which we could easily vendor)

alternatively, we can make sequence a u32 and make the total ordering segment + sequence so that we have u64 in total (sequences are unique within a segment)

I agree 8 bytes is a lot. Another idea I was considering is to add an attribute flag packed into the record type to specify the segment_id size. We could use 4 bytes by default, but switch to 8 bytes if needed. We probably wouldn't need to implement it until we need it. Just need to know it is possible.

I do like the idea to consider segment + sequence as a composite. One thing that bothers me a little bit is that we would have to bump when the u32 sequence is exhausted. Perhaps that's not a big deal, but I did kind of like the idea of representing "semantic segments" as we have with timeseries (each bucket represents an hour of time). It would be annoying to have to handle overflow segments. Perhaps we could consider using varlength u64 for the sequence number which always resets to 0? So u32 segment_id and varlength u64 sequence. This option wasn't possible until we had the fancypants delimiter for the key.

We could use 4 bytes by default, but switch to 8 bytes if needed.
So u32 segment_id and varlength u64 sequence

I like both of these ideas! I agree that dealing with overflow-triggering segment bumps is not fun.

Is the concern the storage overhead? I would imagine that gets optimized away pretty well by the prefix encoding in the SSTs

that + memory overhead, I guess prefix compression would basically eliminate storage overhead.

I implemented something close to what we discussed. First, I switched to a u32 segment encoding, Second, I changed the sequence number to be a varlength u64 which is defined relative to the base sequence from the SegmentMetadata.

that's perfect since we'll get great prefix compression from the segment in slate but actually pretty bad compression from the sequence number so this may turn out to be better overall than the original design 👍

agavra · 2026-01-08T00:03:55Z

+
+1. **On open**: When a new segment is created, a `SegmentMeta` record is written with `start_seq`, `start_time_ms`, and empty `user_meta`. This ensures the segment is immediately discoverable.
+
+2. **On seal**: When `seal_segment()` is called with user metadata, the `SegmentMeta` record is overwritten to include the user-provided bytes. If no user metadata is provided, the record is left unchanged.


we probably need to track the current segment id and increment it somewhere, right?

I think we can just read the metadata in reverse order to find the last segment.

I kind of expected we would keep the segment metadata in memory. It's tiny.

makes sense, I had a brain fart that we could just compute it from the latest SST last key on restart

rodesai

This makes sense to me. One high level thought is whether the notion of a segment is something we should just bake into slatedb directly. It feels very similar to what we'd want with twcs. Need to think more about what that would look like.

rodesai · 2026-01-08T17:09:38Z

+
+```
+Log Entry:
+  | version (u8) | type (u8) | segment_id (u64 BE) | key (TerminatedBytes) | sequence (u64 BE) |


Is the concern the storage overhead? I would imagine that gets optimized away pretty well by the prefix encoding in the SSTs

rodesai · 2026-01-08T17:17:03Z

+
+```
+SegmentMeta Record:
+  Key:   | version (u8) | type (u8=0x03) | segment_id (u64 BE) |


one thought: does it make sense to include the notion of a segment key and a segment id, and use the id as the key for internally generated segments? The idea being that when we support user-generated segments they can select their own keys that can be looked up later

That's a nice idea. Do you think this could be added through a version bump later? Another interesting idea I was considering is to allow segments to be defined at prefix/range granularity. That might open the door to enforcing different retention semantics for different keys. I do think there might an interesting application design space here.

I think these are neat ideas but I'm +1 to keeping things as simple as possible and only tinroducing those concepts if necessary later to avoid complicating things

rodesai · 2026-01-08T17:17:48Z

+
+1. **On open**: When a new segment is created, a `SegmentMeta` record is written with `start_seq`, `start_time_ms`, and empty `user_meta`. This ensures the segment is immediately discoverable.
+
+2. **On seal**: When `seal_segment()` is called with user metadata, the `SegmentMeta` record is overwritten to include the user-provided bytes. If no user metadata is provided, the record is left unchanged.


I think we can just read the metadata in reverse order to find the last segment.

hachikuji · 2026-01-09T00:57:10Z

One high level thought is whether the notion of a segment is something we should just bake into slatedb directly. It feels very similar to what we'd want with twcs.

@rodesai I agree that is the interesting question. I was wondering if the API in slatedb could work with minimal assumptions about key structure. For example, maybe we could have an API to mark a range of keys as "sealed" which makes those keys immutable and stops compaction? I think we need a design in slate sooner than later. Not sure this work needs to block behind it though?

rodesai · 2026-01-10T00:12:09Z

Not sure this work needs to block behind it though?

Totally agree. We can move forward with the approach here and revisit as we understand the problem better.

hachikuji · 2026-01-12T18:11:50Z

If there are no objections, I'll merge this.

rodesai

LGTM

RFC for log segmentation

6110030

agavra reviewed Jan 8, 2026

View reviewed changes

rodesai reviewed Jan 8, 2026

View reviewed changes

This was referenced Jan 8, 2026

[log] Design log retention semantics #38

Open

[log] RFC for range-based log scanning #40

Open

hachikuji added 6 commits January 8, 2026 14:46

Merge remote-tracking branch 'origin/main' into log-segment-rfc

06f6e3d

Move public api to potential future work

b8ee4e6

Consolidate future work

50e0bea

try to make motivation clearer

2d9591b

More compact segment representation

31f14c6

remove unclear checkpoint reference

bd44af0

rodesai approved these changes Jan 12, 2026

View reviewed changes

hachikuji merged commit 4a84782 into main Jan 12, 2026
1 check passed

hachikuji deleted the log-segment-rfc branch January 12, 2026 18:41


		1. On open: When a new segment is created, a `SegmentMeta` record is written with `start_seq`, `start_time_ms`, and empty `user_meta`. This ensures the segment is immediately discoverable.

		2. On seal: When `seal_segment()` is called with user metadata, the `SegmentMeta` record is overwritten to include the user-provided bytes. If no user metadata is provided, the record is left unchanged.

Uh oh!

Conversation

hachikuji commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

agavra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hachikuji Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hachikuji Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rodesai left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hachikuji commented Jan 9, 2026

Uh oh!

rodesai commented Jan 10, 2026

Uh oh!

hachikuji commented Jan 12, 2026

Uh oh!

rodesai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hachikuji commented Jan 7, 2026 •

edited

Loading

hachikuji Jan 8, 2026 •

edited

Loading

hachikuji Jan 8, 2026 •

edited

Loading