feat(snapshot): make pt6 odin partition cronjob viable (SSD store split, R2-direct upload, rclone concurrency)#3495
Merged
Conversation
…it, R2-direct upload, rclone concurrency)
The pt6 odin snapshot CronJob was effectively broken: create-snapshot's
CopyStates is unusably slow with the RocksDB store on HDD, the upload hangs
on the AWS S3 GLACIER archive step (on-prem -> AWS large multipart PUTs), and
even R2 uploads run at ~0.6MB/s with rclone's default single-stream window.
The weekly snapshot was manually published on 2026-06-28 after working around
all three; this bakes that working setup into the chart so the cronjob (and
the daily preload) can run unattended.
Changes (all backward-compatible; other networks unchanged):
- _snapshot_partition_helpers.tpl: hybrid volumeMode now accepts
chainDataVolume.hostPath (not just a PVC) and an optional txVolume.hostPath
submounted at /data/headless/tx. Lets the store live on SSD except the bulk
`tx` CF (268G on HDD), keeping the random-I/O-heavy `states` on SSD where
CopyStates is tolerable. new_states is created on the SSD store root (a real
dir, so create-snapshot's Directory.Delete works — a separate mount would
fail with "Device or resource busy").
- partition/upload_snapshot.sh:
* snapshot.archiveToS3 (default true): when false, skip the S3 GLACIER
archive and upload to R2 directly from the local files.
* snapshot.rcloneUploadConcurrency (default 4): parallel chunk streams.
* snapshot.uploadSplitParts (default true): when false, skip re-uploading
>4G files as .partNNN chunks (download_snapshot.sh falls back to the whole
file, so parts are an optimisation, not a requirement).
* Each artifact crosses the WAN only once (upload once, then server-side
copy to the other R2/internal paths) instead of re-uploading per dest.
* metadata/latest.json is published LAST, after the zip+state+parts, so the
public pointer never references a not-yet-uploaded state_latest.zip.
- 9c-pt6/network/odin.yaml: chainDataVolume.hostPath=/opt/odin-store +
txVolume.hostPath=/data/snapshot-chain-data/tx; archiveToS3=false,
rcloneUploadConcurrency=16, uploadSplitParts=false.
Validated: `helm template` renders pt6 (hostPath store + tx submount + tuning
values) and heimdall (unchanged /output layout, defaults preserved); rendered
upload_snapshot.sh passes `bash -n`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
배경
pt6 odin 스냅샷 CronJob이 사실상 동작 불가였음 (공개 latest.json 5/26 동결). 2026-06-28 수동으로 6/28 스냅샷(#18815070) 발행하며 3가지 병목을 우회했고, 이 PR은 그 동작 구성을 차트에 반영해 주간 partition + 일일 preload 크론잡이 무인 실행되도록 함.
3가지 병목:
변경 (전부 하위호환 — 다른 네트워크 영향 없음)
_snapshot_partition_helpers.tplhybrid
volumeMode가 이제:chainDataVolume.hostPath지원 (PVCname뿐 아니라 host dir)txVolume.hostPath→/data/headless/tx로 submount→ store를 SSD에 두되 덩치
txCF(268G)만 HDD에. 랜덤 I/O 많은states가 SSD에 있어 CopyStates 감내 가능. new_states는 SSD store 루트의 일반 디렉토리로 생성됨(별도 마운트면 create의Directory.Delete가 "Device or resource busy"로 실패).partition/upload_snapshot.shsnapshot.archiveToS3(기본 true): false면 S3 GLACIER archive 건너뛰고 R2에 로컬에서 직접 업로드snapshot.rcloneUploadConcurrency(기본 4): 병렬 청크 스트림 (pt6 16 → ~3.4MB/s)snapshot.uploadSplitParts(기본 true): false면 >4G 파일의.partNNN재업로드 생략 (download_snapshot.sh가 whole 파일로 폴백)values.yaml위 3개 키 기본값 추가 (archiveToS3:true, rcloneUploadConcurrency:4, uploadSplitParts:true) + hybrid hostPath/txVolume 주석.
9c-pt6/network/odin.yamlchainDataVolume.hostPath=/opt/odin-store+txVolume.hostPath=/data/snapshot-chain-data/tx;archiveToS3=false,rcloneUploadConcurrency=16,uploadSplitParts=false.검증
helm template: pt6 렌더(hostPath store + tx submount + 튜닝값) + heimdall 렌더(기존/output레이아웃·기본값 유지, 회귀 없음) 확인upload_snapshot.sh→bash -n통과/opt/odin-store,/data/snapshot-chain-data/tx)이 존재해야 함 (이번 인시던트로 이미 존재).🤖 Generated with Claude Code