Skip to content

Impl a streaming writer temporary file storage abstraction#2175

Open
mengpromax wants to merge 4 commits intoqax-os:masterfrom
mengpromax:feat/rawDataAbstraction
Open

Impl a streaming writer temporary file storage abstraction#2175
mengpromax wants to merge 4 commits intoqax-os:masterfrom
mengpromax:feat/rawDataAbstraction

Conversation

@mengpromax
Copy link
Copy Markdown
Contributor

PR Details

impl a streaming writer temporary file storage abstraction to support custom temporary file storage.

Description

  1. replace streamingWriter.rawData with a interface.
  2. add a chunk size field to buffered writer which is default to 16MB or can be set by user.

Related Issue

  1. This will close StreamChunkSize(16MB) variable setting function #1927, will support buffered writer custom chunk size.

Motivation and Context

This issue #1927 inspire me.

How Has This Been Tested

  1. pass all exist UT.
  2. add custom temporary file storage UT.

Types of changes

  • Docs change / refactoring / dependency upgrade
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

…support more storage

Change-Id: Ibaca66b80a3ed8793396e1386bf7c0b2db7c21c0
Signed-off-by: mengzhongyuan <mengzhongyuan@bytedance.com>
@mengpromax mengpromax force-pushed the feat/rawDataAbstraction branch from d46c7b5 to 01b1854 Compare July 11, 2025 15:31
@xuri xuri added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jul 13, 2025
mengpromax and others added 2 commits February 8, 2026 18:23
Change-Id: I7f207e3de005574f79163b3a7b8c68c20ca9adad
Signed-off-by: mengzhongyuan <mengzhongyuan@bytedance.com>
@mengpromax
Copy link
Copy Markdown
Contributor Author

@xuri Please help remove the wip and review it, this pr also resolves #2178.

@xuri xuri changed the title WIP: impl a streaming writer temporary file storage abstraction Impl a streaming writer temporary file storage abstraction Feb 8, 2026
@codecov
Copy link
Copy Markdown

codecov bot commented Feb 8, 2026

Codecov Report

❌ Patch coverage is 76.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 99.54%. Comparing base (4917cff) to head (3651ed2).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
lib.go 42.85% 2 Missing and 2 partials ⚠️
stream.go 88.88% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2175      +/-   ##
==========================================
- Coverage   99.56%   99.54%   -0.03%     
==========================================
  Files          32       32              
  Lines       26291    26307      +16     
==========================================
+ Hits        26176    26186      +10     
- Misses         60       63       +3     
- Partials       55       58       +3     
Flag Coverage Δ
unittests 99.54% <76.00%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Change-Id: I04fc04bedb438b3cb692b43c8114a56cc114bb2c
Signed-off-by: mengzhongyuan <mengzhongyuan@bytedance.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

StreamChunkSize(16MB) variable setting function

2 participants