Release oasis storage prune and compact subcomands

[martintomazic]

Closes #6519.

Relatively trivial to review. LOC is big because I extracted commands to separate files and added missing tests (separate commits).

Motivation

1. Offline pruning/compaction was introduced because late pruning 1. may not reclaim disk space, 2. may cause node to fall behind even once registered as ready (pending compaction load).
2. We could also use it to create go/oasis-node: Enable snapshot creation with exact start version #6423, serving as alternative to go/oasis-node/cmd/storage: Add create and import checkpoint cmd #6454.

How to test locally

For testing I used latest Sapphire snapshot (cca 7 months of data, keep_n = 100_000). It took 10min to prune and 6min to compact. For consensus I used snapshot-old (cca 3 months of data, keep_n = 100_000), due to limited disk space on my machine. Pruning took 2min and compaction 3min.

Node started syncing normally after prune/compact commands.

Possible follow-up

For #6454, we may want to add additional flag to the pruning command, e.g. retain_height, that ignores pruning config and calculates corresponding rounds at this height of every runtime, implementing keep_from proposal.

[netlify]

✅ Deploy Preview for oasisprotocol-oasis-core canceled.

Name	Link
🔨 Latest commit	d3189ce
🔍 Latest deploy log	https://app.netlify.com/projects/oasisprotocol-oasis-core/deploys/6a26b7c552018a000878ffbd

[martintomazic]

In practice we should probably also respect executor prune handler that should prevent pruning past last normal round.

// runtimeLastNormalRound returns the last normal round for the given runtime.
func runtimeLastNormalRound(ctx context.Context, ndb db.NodeDB, runtimeID common.Namespace) (uint64, error) {
	latest, ok := ndb.GetLatestVersion()
	if !ok {
		return 0, fmt.Errorf("consensus node DB is empty")
	}

	roots, err := ndb.GetRootsForVersion(latest)
	if err != nil {
		return 0, fmt.Errorf("failed to get roots for consensus version %d: %w", latest, err)
	}
	if len(roots) == 0 {
		return 0, fmt.Errorf("no roots found for consensus version %d", latest)
	}

	tree := mkvs.NewWithRoot(nil, ndb, roots[0], mkvs.WithoutWriteLog())
	defer tree.Close()

	s := roothashState.NewImmutableState(tree)
	rtState, err := s.RuntimeState(ctx, runtimeID)
	if err != nil {
		return 0, fmt.Errorf("failed to get runtime state: %w", err)
	}

	return rtState.LastNormalRound, nil
}

We have two options:

1. Pass consensus node DB to pruneRuntimeDBs.
   • Or just open and close it there (simplest and least changes).
   • This feels weird as this function should probably accept retainRound, like pruneConsensusDBs should accept retainHeight instead of runtimeIds (solution 2.).
2. Add new consensusRetainHeight and runtimeRetainRound functions that precompute this limits, possibly write unit tests for those two functions. The annoying part is that those two functions consume runtime histories, nodedbs and consensus nodedbs, meaning we loose resource encapsulation and need to keep them open throughout the whole command. Feels like a better direction, but requires a thorough refactor + makes logging and orchestration incredibly messy.

In addition adding this handler requires consensus state to be always present. Without it you can prune runtime state without having a consensus state locally (e.g. when hacking state if imported using snapshots, don't think this is needed though).

[martintomazic]

We can also hardcode sain default e.g. min few days retention, and leave a TODO issue.

[martintomazic]

E.g. this could be func consensusRetainHeight(ndb db.NodeDB, histories []history.Histories)(uint64, bool, error) and this function takes retainHeight: uint64 instead of the last two params, which would also allow us to unit test the business logic. As stated above this complicates the orchestration a lot though :(

[martintomazic]

NIT: We should probably wire

 signal.Notify(sigCh, os.Interrupt, syscall.SIGTERM) .

As cobra context does not capture this signals by default, resulting in non-graceful shutdown (never had an issue so fat though). Still it is annoying that BadgerDB command does not expose cancelation api, so even with this fixed it won't be complete.

[martintomazic]

Ready for a review.

Please check the PR context, the main question is how much do we want to complicate our life with prune handlers - comment.

Since the PR is already rather big, I am open to push one PR infront that does:

• factors prune/compact commands to separate files,
• adds missing tests
• minor fixes
• prune block and state store in small intervals.

Then this PR becomes much smaller and focused only one new runtime logic and releasing the commands.

[codecov]

Codecov Report

❌ Patch coverage is 60.30769% with 129 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.09%. Comparing base (b6a87ba) to head (e5b4a63).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
go/oasis-node/cmd/storage/prune.go	60.47%	45 Missing and 38 partials ⚠️
go/oasis-node/cmd/storage/compact.go	53.65%	19 Missing and 19 partials ⚠️
go/runtime/history/db.go	73.91%	4 Missing and 2 partials ⚠️
go/runtime/history/history.go	75.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6521      +/-   ##
==========================================
- Coverage   64.12%   64.09%   -0.04%     
==========================================
  Files         698      700       +2     
  Lines       68253    68416     +163     
==========================================
+ Hits        43770    43854      +84     
+ Misses      19556    19506      -50     
- Partials     4927     5056     +129     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.