feat(evm): per-network status-check retry interval

[dylankilkenny]

Problem

The relayer checks a pending EVM transaction on a repeating schedule. The first check is configurable per network (#847). Every later check uses a fixed wait of 8–12 seconds (STATUS_EVM_BACKOFF in src/queues/retry_config.rs). The wait is the same for all EVM networks.

On a chain that finalizes in 1 second, the relayer still reports confirmation after ~10 seconds. The fixed wait causes this delay.

Goal

Let each network configure the wait between status checks. Networks that do not set the field keep today's behavior exactly. Real errors (RPC failures) keep the current backoff behavior — this setting only changes how often we re-check a healthy, not-yet-final transaction.

Implementation steps

1. Add the config field

Add status_check_retry_delay_seconds: Option<u64> to EvmNetworkConfig in src/config/config_file/network/evm.rs. Copy the pattern of status_check_initial_delay_seconds from #847: same validation (reject values outside 1–100), same inheritance (child overrides parent).

Carry it on EvmNetwork in src/models/network/evm/network.rs as Option<u64> — do not resolve a default here. An unset value must stay None all the way to the retry code, which then keeps today's backoff exactly. (This differs from the initial-delay field, which resolves to 8. Here a resolved default would silently change the cadence of every EVM network from a growing 8→12s backoff to a fixed 8s.) Check the 1–100 range again in TryFrom<NetworkRepoModel>.

{ "type": "evm", "network": "fast-chain", "status_check_retry_delay_seconds": 1 }

2. Add the value to the job payload

Add a field to TransactionStatusCheck in src/jobs/job.rs:

#[serde(default)]
pub status_retry_delay_seconds: Option<u32>,

Set it wherever the code creates an EVM status-check job. There are three places: process_transaction_request in src/domain/relayer/evm/evm_relayer.rs, the nonce gap-fill in src/domain/relayer/evm/nonce.rs, and the nonce-recovery checks in src/domain/transaction/evm/evm_transaction.rs. Each place already has the network object (or can load it with one repository call).

#[serde(default)] keeps old queued jobs and old binaries working: a missing field means "use current behavior".

3. Mark "not final" with a typed error

Background. The handler cannot ask the queue to "run this check again later". It has one tool: return an error. The queue retries failed jobs, and that retry is the next status check. So when a transaction is not final, the handler returns an error on purpose. As a result, two different situations produce the same error type today (Err(HandlerError::Retry(String)) at src/jobs/handlers/transaction_status_handler.rs:121):

Case A: the check worked; the transaction is just not confirmed yet
        → Err(Retry("transaction status: ... - not in final state, retrying"))
Case B: the check itself failed (RPC down, timeout)
        → Err(Retry("connection refused ..."))

The retry code in steps 4 and 5 must treat these differently:

• Case A → wait the network's configured interval (for example 1 second on a fast chain).
• Case B → keep the exponential backoff. Do not hammer a broken RPC every second.

The retry code only receives the job and the error — it cannot inspect the transaction. So the error itself must say which case occurred. Matching the message string would work but breaks silently if anyone rewords the message. Use a type instead:

#[derive(Debug, thiserror::Error)]
#[error("transaction not in final state")]
pub struct NotYetFinal;

Return it from the not-final branch (new HandlerError variant, Case A only). Update From<HandlerError> in src/queues/worker_types.rs so the marker survives into the queue error. The retry code then checks error.downcast_ref::<NotYetFinal>().is_some().

4. Redis worker: custom retry policy

The EVM status worker sets its retry wait once, at startup (src/queues/redis/worker.rs:432). Replace that policy with a custom tower::retry::Policy (~70 lines; use apalis 0.7.4's BackoffRetryPolicy as the template — the trait gives the policy the full job and the error):

• Error is NotYetFinal and the payload value is valid (1–100) → sleep that many seconds, plus jitter.
• Any other retryable error → keep the existing STATUS_EVM_BACKOFF exponential backoff.
• Keep the attempt counter increment (metrics and logs use it).

Edge cases the policy must handle:

• Error::Abort → do not retry. Same as the stock policy.
• Payload field is None (old queued job) → use the stock exponential backoff. Behavior is identical to today.
• Payload value outside 1–100 (should not occur; validation rejects it) → treat it as absent and use the stock backoff. An invalid value must not select the fastest cadence.
• Out-of-retries → convert the error to Abort, like the stock policy. This branch never runs (retries = usize::MAX) but keep it for safety.
• tower clones the policy once per request session. Per-job backoff state is safe; do not share mutable state across jobs.
• The shared status handler serves all network types. It returns NotYetFinal for every network. This is safe: only the EVM worker gets the new policy; the Stellar and generic workers treat NotYetFinal like any error and back off as today.

5. Other queue backends

SQS, RabbitMQ, and Pub/Sub already compute the retry wait per message by reading the job body. Extend that code to prefer the new payload field when the error is NotYetFinal:

• Shared path for RabbitMQ/Pub/Sub: compute_status_retry_delay in src/queues/worker_shared.rs.
• SQS has its own copy: compute_status_retry_delay in src/queues/sqs/worker.rs. Unify the two copies first if that is cheap.

Tests

• Config: reject 0 and 101; accept 1 and 100; inheritance; an unset field stays None on the model.
• Payload: a job without the new field still deserializes.
• Redis policy: NotYetFinal → sleep equals the payload value; other errors → exponential backoff.
• Integration: network with status_check_retry_delay_seconds: 2 → checks run ~2 seconds apart until the transaction is final.

Notes

• The value is fixed when the transaction is created. A config change applies to new transactions only. Acceptable: transactions live for minutes.
• EVM only. Stellar and Solana keep their current behavior.
• refactor(queues): self-scheduling status checks — remove the pending-tx concurrency ceiling #871 (self-scheduling checks) replaces step 4 when it lands. Steps 1–3 and 5 carry forward unchanged.
• Follow-up idea (not in scope): derive the default from average_blocktime_ms and keep this field as the override.

Size: ~250–350 lines across ~6 files plus tests. No infrastructure changes. Safe to roll back.