Handle full sync differently on the main branch

[gustavovalverde]

GitHub changed the behavior of the concurrency, showing cancellation statuses as failures, rather than ignoring them and showing it as successes. Having all those ❌ in the main branch is confusing, and we should decide how to handle this situation.

We have several options (including others not listed here):

• Not running the full sync on each merge, as we're anyways cancelling those until a previous one finishes
• Have it on its own workflow and have a more specific decision tree on when to trigger it
• Schedule it to run twice a week or weekly (as it anyways takes 3+ days to complete)
• Having a trigger on the PR, as we might want to validate if a critical PR runs a Full sync safely before merging it (required check for specific labels [?])

[teor2345]

GitHub changed the behavior of the concurrency, showing cancellation statuses as failures, rather than ignoring them and showing it as successes. Having all those ❌ in the main branch is confusing, and we should decide how to handle this situation.

It also makes checking for main branch failures really difficult, because every failed or cancelled job in every main branch PR needs to be checked individually.

* Schedule it to run twice a week or weekly (as it anyways takes 3+ days to complete)

I suggest we run it weekly at the end of each week.

Until we can get failure notifications on Slack or Discord, the person on main branch failures will need to check the scheduled task for failures as well. (Unfortunately!) If it helps, we can get a report back from them at each engineering sync.

Would this change be blocked by fixing the full sync timeouts in the current CI? (I don't think it has to be.)

[teor2345]

Also, as of PR #6581, we're running a testnet full sync on each merge to main. So we will still get some kind of sync status on the main branch.

The differences are:

• the testnet chain has fewer transactions, and doesn't contain real-world data
• testnet can be less reliable than mainnet (but it has passed every time we've run it)
• testnet syncs in 2 hours

[mpguerra]

@gustavovalverde this is quite an old discussion now and I believe we already implemented a number of mitigations for this behaviour.

Can this discussion be closed now?