Skip to content

feat: implement blob streaming in the aggregator by slivers#2801

Merged
wbbradley merged 4 commits into
mainfrom
streaming-slivers
Jan 15, 2026
Merged

feat: implement blob streaming in the aggregator by slivers#2801
wbbradley merged 4 commits into
mainfrom
streaming-slivers

Conversation

@wbbradley

Copy link
Copy Markdown
Contributor

Description

Implement streaming blobs by slivers via a new endpoint /v1/blobs/{blob_id}/stream.

Test plan

Manually tested. CI Pipeline.


Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that
a user might notice and any actions they must take to implement updates. (Add release notes after the colon for each item)

  • Storage node:
  • Aggregator: There is a new alpha endpoint /v1alpha/blobs/{blob_id}/stream which is intended to reduce latency to first byte, and is intended to be used to stream blob data.
  • Publisher:
  • CLI:

@wbbradley wbbradley force-pushed the streaming-slivers branch 2 times, most recently from 9a67517 to a18a119 Compare December 19, 2025 20:52
@github-actions

Copy link
Copy Markdown
Contributor

Warning: This PR modifies one of the example config files. Please consider the
following:

  • Make sure the changes are backwards compatible with the current configuration.
  • Make sure any added parameters follow the conventions of the existing parameters; in
    particular, durations should take seconds or milliseconds using the naming convention
    _secs or _millis, respectively.
  • If there are added optional parameter sections, it should be possible to specify them
    partially. A useful pattern there is to implement Default for the struct and derive
    #[serde(default)] on it, see BlobRecoveryConfig as an example.
  • You may need to update the documentation to reflect the changes.

@wbbradley wbbradley force-pushed the streaming-slivers branch 2 times, most recently from 3d6668f to 4de2dc4 Compare December 20, 2025 17:06
Comment thread crates/walrus-sdk/src/client/streaming_read_client.rs Outdated
@wbbradley wbbradley force-pushed the streaming-slivers branch 5 times, most recently from fc5857a to 7ad8b46 Compare December 28, 2025 22:40
@wbbradley wbbradley force-pushed the streaming-slivers branch 4 times, most recently from 7b33c91 to 2598d63 Compare January 2, 2026 23:34
@jpcsmith jpcsmith removed their request for review January 5, 2026 11:59
@jpcsmith

jpcsmith commented Jan 5, 2026

Copy link
Copy Markdown
Contributor

Removed myself since Markus is having a look. Let me know if there something that I should look at regardless.

@mlegner mlegner removed their request for review January 6, 2026 16:14
@wbbradley wbbradley force-pushed the streaming-slivers branch 2 times, most recently from 9529114 to 7b9ff3d Compare January 7, 2026 04:38

@halfprice halfprice left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @wbbradley for implementing this feature! This looks great and I only have some minor comments.

Another thing we discussed was to use streaming blob as default and replace the existing blocking get_blob. Can you create a issue tracking this?

Comment thread crates/walrus-service/src/client/daemon/routes.rs Outdated
Comment thread crates/walrus-service/src/client/daemon/routes.rs
Comment thread crates/walrus-service/src/client/daemon/routes.rs Outdated
Comment thread crates/walrus-service/src/client/daemon/routes.rs Outdated
Comment thread crates/walrus-sdk/src/client/streaming.rs Outdated
Comment thread crates/walrus-sdk/src/client/streaming.rs
Comment thread crates/walrus-sdk/src/client/streaming.rs Outdated
Comment thread crates/walrus-sdk/src/client/streaming.rs Outdated
@wbbradley wbbradley force-pushed the streaming-slivers branch 2 times, most recently from fc99dc3 to 83136c6 Compare January 13, 2026 19:07
@wbbradley wbbradley requested a review from halfprice January 14, 2026 04:31
@wbbradley

Copy link
Copy Markdown
Contributor Author

@halfprice I believe I've addressed your comments. I've also implemented a new e2e test called test_streaming_blob.

@wbbradley

Copy link
Copy Markdown
Contributor Author

I've added WAL-1107 to track using this functionality for get_blob.

@halfprice halfprice left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Thanks @wbbradley for adding this.

Comment thread crates/walrus-e2e-tests/tests/test_client.rs
@wbbradley wbbradley merged commit e3a5ab6 into main Jan 15, 2026
24 checks passed
@wbbradley wbbradley deleted the streaming-slivers branch January 15, 2026 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants