fix(lean): reject forward-sync blocks ahead of wall-clock to stop on_tick freeze

[zemse]

What was wrong?

Ream node freezes when a malicious block is received while syncing.

During forward sync, a ream node receives a structurally valid SignedBlock but its slot is very large (e.g. u64::MAX - 5 to match the right proposer slot), the node gets stuck due to the on_tick function. Also the tick is advanced to a large number hence it cannot sync, the db needs to be cleared.

This bug is similar to the one already fixed in #1501. That PR added check in the on_block function to deny such malicious block during live sync. However during forward sync, the on_tick function is called before on_block that would have denied such a payload.

When having multiple peers, if enough peers are malicious then the malicious payload wins. Also a live synced node can potentially backfill and get frozen.

How was it fixed?

The fix is added in the ForwardBackgroundSyncer::start() just before the on_tick invocation. Based on the system time/wall_clock we compute the expected slot and reject any slot that is beyond wall_clock_slot + 1.

How is this tested?

Local tests pass. There are no hive tests for forward syncing. I have done bunch of devnet runs.

Master branch

• 1 lagging + 1 adversary: the lagging node freezes on fwd sync
• 1 lagging + 1 live + 1 adversary: lagging node freezes on fwd sync and even live freezes due to backfill
• 1 lagging + 2 live + 1 adversary: all honest nodes survive

PR branch

• 1 lagging + 1 adversary: no freeze (but no other source to fwd sync)
• 1 lagging + 1 live + 1 adversary: honest nodes survive (except lagging node follows head but finalisation stuck due to adversary quorum issue)
• 1 lagging + 2 live + 1 adversary: all honest nodes survive

To-Do

• I have read CONTRIBUTING.md.
• This PR title follows conventional commits.

[KolbyML]

looks good