Skip to content

fix: improve f3 snapshot import performance with batching#1046

Merged
Kubuxu merged 2 commits into
filecoin-project:mainfrom
hanabi1224:f3-snap-import-perf
Aug 1, 2025
Merged

fix: improve f3 snapshot import performance with batching#1046
Kubuxu merged 2 commits into
filecoin-project:mainfrom
hanabi1224:f3-snap-import-perf

Conversation

@hanabi1224

Copy link
Copy Markdown
Contributor

This PR improves performance of ImportSnapshotToDatastore by

  • changing ds datastore.Datastore to ds datastore.Batching
  • using autobatch internally

Stats,
Loading a ~300MiB Filecoin calibnet F3 snapshot on my laptop takes

  • ~5s into a memory blockstore
  • 30min+ into leveldb without batching
  • ~12s into leveldb with batching

@github-project-automation github-project-automation Bot moved this to Todo in F3 Aug 1, 2025
@hanabi1224 hanabi1224 marked this pull request as ready for review August 1, 2025 10:27
@codecov

codecov Bot commented Aug 1, 2025

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.08%. Comparing base (38f1620) to head (76a0430).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1046      +/-   ##
==========================================
+ Coverage   64.75%   65.08%   +0.33%     
==========================================
  Files          81       81              
  Lines        9894     9896       +2     
==========================================
+ Hits         6407     6441      +34     
+ Misses       2966     2933      -33     
- Partials      521      522       +1     
Files with missing lines Coverage Δ
certstore/snapshot.go 47.82% <100.00%> (+0.92%) ⬆️

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-project-automation github-project-automation Bot moved this from Todo to In review in F3 Aug 1, 2025
@Kubuxu Kubuxu merged commit 9288fba into filecoin-project:main Aug 1, 2025
12 checks passed
@github-project-automation github-project-automation Bot moved this from In review to Done in F3 Aug 1, 2025
@hanabi1224 hanabi1224 deleted the f3-snap-import-perf branch August 1, 2025 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants