Switch trigger timer on p28 boundary (#5350)

# Description

This enables the new trigger timer on the protocol 28 boundary. While we
had discussed the potential for this to be a non-protocol upgrade, and
just ask operators to switch at around the same time, we've received
slow response times historically from tier 1, which could leave the
network in a degraded state. This is the safest way to land the change.

I've also added a flag that, when set, forces validators to use the old
timer after the p28 upgrade. In the event that we see an unexpected perf
issue after the upgrade, we can instruct operators to set this flag to
roll back the change without a protocol upgrade. Finally, I also added
an upgrade path test, and fixed some flakey asserts. Specifically, in
the node drift tests, we assert that the drifting node detects it's
drift and falls back to the old timer. However, if the drifting node's
timer is externalized as the closeTime, the non-drifting nodes fall
back, as they think they are drifting relative to the rest of the
network. This is safe behavior, but the test would sometimes fail based
on leader schedules.

# Checklist
- [x] Reviewed the
[contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes)
document
- [x] Rebased on top of master (no merge commits)
- [x] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio
extension)
- [x] Compiles
- [x] Ran all tests
- [ ] If change impacts performance, include supporting evidence per the
[performance
document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)

Author: SirTyson

docs/metrics.md

@@ -201,7 +201,7 @@ scp.timing.externalized                   | timer     | time spent in ballot pro
 scp.timing.first-to-self-externalize-lag  | timer     | delay between first externalize message and local node externalizing
 scp.timing.self-to-others-externalize-lag | timer     | delay between local node externalizing and later externalize messages from other nodes
 scp.timing.ballot-blocked-on-txset        | timer     | time balloting was blocked waiting for a txset download (milliseconds)
-scp.trigger.prepare-start-fallback        | meter     | experimental trigger timer fell back from the network-close-time anchor to the local prepare-start anchor
+scp.trigger.prepare-start-fallback        | meter     | trigger timer fell back from the network-close-time anchor to the local prepare-start anchor
 scp.value.invalid                         | meter     | SCP value is invalid
 scp.value.valid                           | meter     | SCP value is valid
 scp.slot.values-referenced                | histogram | number of values referenced per consensus round

src/herder/HerderImpl.cpp

@@ -1447,8 +1447,18 @@ HerderImpl::setupTriggerNextLedger()
 
     auto now = mApp.getClock().now();
 
+    // Starting from protocol 28 the trigger timer is anchored on the
+    // network-agreed consensus close time.
+    // FORCE_OLD_STYLE_PREPARE_START_TRIGGER_TIMER forces the older
+    // prepare-start anchor as an emergency fallback.
+    bool const useConsensusCloseTimeAnchor =
+        protocolVersionStartsFrom(
+            lcl.header.ledgerVersion,
+            CONSENSUS_CLOSE_TIME_TRIGGER_PROTOCOL_VERSION) &&
+        !mApp.getConfig().FORCE_OLD_STYLE_PREPARE_START_TRIGGER_TIMER;
+
     auto lastLedgerStartingPoint =
-        mApp.getConfig().EXPERIMENTAL_TRIGGER_TIMER
+        useConsensusCloseTimeAnchor
             ? triggerAnchorFromConsensusCloseTime(lastIndex, now, milliseconds)
             : triggerAnchorFromPrepareStart(lastIndex, now, milliseconds);
 

src/herder/HerderImpl.h

@@ -358,7 +358,7 @@ class HerderImpl : public Herder
         medida::Meter& mEnvelopeValidSig;
         medida::Meter& mEnvelopeInvalidSig;
 
-        // Marked when the experimental trigger timer falls back from the
+        // Marked when the trigger timer falls back from the
         // network-close-time anchor to the local prepare-start anchor.
         medida::Meter& mTriggerPrepareStartFallback;