Flaky CI Failure: test_hmac_and_consent_preference_sync times out waiting for ConsentReceived

[github-actions]

Flaky CI Failure: test_hmac_and_consent_preference_sync times out waiting for ConsentReceived

Workflow: Test
Failed run: https://github.com/xmtp/libxmtp/actions/runs/27215352372
Commit: c654136
Failed jobs: test-workspace / Test (Rust Workspace)

Summary

The Rust test worker::device_sync::tests::test_hmac_and_consent_preference_sync in xmtp_mls failed because it timed out waiting for the ConsentReceived metric to reach a count of 2. The worker only processed 1 consent record within the allotted timeout, causing an Expired error at crates/xmtp_mls/src/worker/device_sync/tests.rs:298. This test exercises HMAC-based device sync for consent preferences across two clients.

Error Details

2026-06-09T15:25:47.530326Z  INFO crates/xmtp_mls/src/worker/metrics.rs:39:
  [c5a7e153...] successfully waited for ConsentReceived, Err(Expired)

2026-06-09T15:25:47.530335Z ERROR crates/xmtp_mls/src/worker/metrics.rs:47:
  Timed out waiting for ConsentReceived to be >= 2. Value: 1

thread 'worker::device_sync::tests::test_hmac_and_consent_preference_sync' (50772) panicked at crates/xmtp_mls/src/worker/device_sync/tests.rs:298:15:
called `Result::unwrap()` on an `Err` value: Expired
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 635 filtered out; finished in 10.33s

Analysis

The test sets up two clients and syncs consent preferences between them using HMAC-based device sync, then asserts that ConsentReceived reaches at least 2 (one per client). Only 1 consent record arrived within the timeout — suggesting the second client's consent sync message was either delayed, dropped, or not delivered before the deadline.

This same test appeared as a flaky-but-passing test (FLAKY 2/4) in prior runs (e.g., 2026-05-18 recurrence on issue #3438), indicating it can fail on the first attempt but recover. In this run it failed hard across all retries after taking ~10s. The concurrent failure of client::tests::should_reconnect (network ConnectionReset error) in the same run suggests possible backend or network instability contributed to the consent sync message not being delivered in time.

This is a test/code boundary failure — the timeout may be too tight for CI environment variability, or the underlying consent sync delivery path has a race condition under load.

---

Reported by Flaky Failure Watcher

[xmtp-coder-tasks]

Task is running: https://sandbox.xmtp.team/tasks/insipx/c769608e-00fc-4974-ad45-876b9139827c

[insipx]

Bugfix: flaky test_hmac_and_consent_preference_sync timeout on ConsentReceived

Issue: #3749

Current Behavior

worker::device_sync::tests::test_hmac_and_consent_preference_sync (in
crates/xmtp_mls/src/worker/device_sync/tests.rs) intermittently times out
waiting for the ConsentReceived metric to reach 2, panicking with Err(Expired)
at the register_interest(SyncMetric::ConsentReceived, 2).wait() call.

Root cause is a missing synchronization barrier in the test, not a product bug.
The second consent phase does:

alix1_group.update_consent_state(ConsentState::Allowed)?;   // async publish by alix1's worker
alix2.sync_all_welcomes_and_groups(None).await?;            // one-shot pull
alix2.worker().register_interest(SyncMetric::ConsentReceived, 2).wait().await?;

update_consent_state only queues work for alix1's device-sync worker, which
publishes the consent update to the sync group asynchronously (incrementing
ConsentSent). Meanwhile register_interest(...).wait() waits passively on the
receiver metric — it does not itself drive any re-sync. So alix2's single
sync_all_welcomes_and_groups call is the only opportunity to pull the second
consent. sync_all_welcomes_and_groups filters out groups with no new server
activity (filter_groups_needing_sync); if alix2 syncs before alix1's consent
message has landed on the server, the device-sync group is filtered out, alix2
never pulls the update, and ConsentReceived stays at 1 until the 10s timeout
fires.

This is a classic race: it passes when alix1 happens to publish before alix2's
sync, and fails otherwise (previously reported FLAKY 2/4).

Note: the first consent phase in the same test already guards against this by
waiting for ConsentSent before the receiver syncs — the second phase simply
omitted the same barrier.

Expected Behavior

The test deterministically waits for alix1 to publish the second consent update
(ConsentSent) before alix2 performs its pull sync, mirroring the existing
first-consent barrier. After the barrier, alix2's sync is guaranteed to see the
consent message on the server, so ConsentReceived reliably reaches 2 and the
test passes on every run.

Fix (test-only), inserted before alix2's sync:

alix1.worker().clear_metric(SyncMetric::ConsentSent);
alix1_group.update_consent_state(ConsentState::Allowed)?;
alix1.worker().register_interest(SyncMetric::ConsentSent, 1).wait().await?;

alix2.sync_all_welcomes_and_groups(None).await?;

clear_metric is called before update_consent_state (as in the first-consent
phase) so the barrier waits on the Allowed publish specifically, with no race
between clear and increment.

Unchanged Behavior

• Production code is untouched; the change is confined to the test body.
• The assertions the test makes (alix2_group.consent_state() == Allowed, HMAC
  key equality, first-consent Denied propagation) are unchanged.
• No change to the device-sync worker, metrics, or sync APIs.
• Other tests in the suite are unaffected.