Skip to content

Fix racy CurrentEventsByTag_should_find_existing_events test - #76

Open
Aaronontheweb wants to merge 2 commits into
akkadotnet:devfrom
Aaronontheweb:fix/racy-current-events-by-tag-spec
Open

Fix racy CurrentEventsByTag_should_find_existing_events test#76
Aaronontheweb wants to merge 2 commits into
akkadotnet:devfrom
Aaronontheweb:fix/racy-current-events-by-tag-spec

Conversation

@Aaronontheweb

Copy link
Copy Markdown
Member

Summary

  • Override ReadJournal_query_CurrentEventsByTag_should_find_existing_events from the base TCK CurrentEventsByTagSpec to fix a flaky test caused by EventStore's eventually consistent projections
  • The base test uses backpressure paging (Request(2), ExpectNoMsg, Request(2)) against a projected tag stream, but OnComplete races with ExpectNoMsg when the projection hasn't indexed all events yet
  • The override adds a 300ms delay for projection catch-up and requests all events at once, following the same pattern already used by ReadJournal_query_CurrentEventsByTag_should_see_all_150_events

Root Cause

The base TCK test writes 6 events (3 tagged "green"), then immediately queries CurrentEventsByTag("green", NoOffset). It requests only 2 items, expects them, then calls ExpectNoMsg(500ms) before requesting more. However, EventStore projections are eventually consistent -- the projected tag stream may only contain the first 2 green events when queried, causing the stream to complete immediately and OnComplete to arrive during the ExpectNoMsg window (observed arriving after just 36 nanoseconds).

Test plan

  • Verify the overridden test passes consistently in CI (no more flaky OnComplete during ExpectNoMsg)
  • Verify the test still validates the same core behavior: querying "green", "black", and "apple" tags returns the correct events with correct persistence IDs, sequence numbers, and payloads

Override the base TCK test to account for EventStore's eventually
consistent projections. The base test uses backpressure (Request 2,
ExpectNoMsg, Request 2) to page through a tagged stream, but the
EventStore projection may complete before all events are indexed,
causing OnComplete to race with ExpectNoMsg. The override adds a
300ms delay for projection catch-up and requests all events at once,
consistent with the existing override pattern used by
ReadJournal_query_CurrentEventsByTag_should_see_all_150_events.
@Aaronontheweb
Aaronontheweb enabled auto-merge (squash) March 17, 2026 17:01
@Aaronontheweb
Aaronontheweb disabled auto-merge March 17, 2026 17:01
… catch-up

EventStore projections are eventually consistent. Instead of sleeping for
an arbitrary duration and hoping the projection has caught up, poll the
projection using AwaitConditionAsync until the expected number of events
are indexed. This eliminates the race condition that caused flaky failures
in CurrentEventsByTag_should_find_existing_events.

Also fixes the same non-deterministic pattern in the 150-event test and
the offset exclusivity test.
Aaronontheweb added a commit to Aaronontheweb/akka.net that referenced this pull request Mar 17, 2026
Add a protected virtual extensibility point to CurrentAllEventsSpec,
CurrentEventsByTagSpec, and CurrentEventsByPersistenceIdSpec that is
called after events are written and before queries are executed.

Backends with eventually-consistent read models (EventStore, Kafka,
etc.) can override this method to wait for their read side to catch up,
instead of being forced to override entire TCK test methods with
Thread.Sleep hacks.

Evidence: akkadotnet/Akka.Persistence.EventStore#76
Aaronontheweb added a commit to Aaronontheweb/akka.net that referenced this pull request Mar 17, 2026
Replace the virtual WaitForReadSideAsync() hook with inline polling
using AwaitConditionAsync. Each test now polls its respective Current*
query until the expected number of events are indexed before running
assertions. This is deterministic: synchronous backends pass on the
first poll, eventually-consistent backends converge.

Fixes the root cause of flaky TCK failures in backends like EventStore
where projections are eventually consistent. Previously, these backends
were forced to override entire test methods with Thread.Sleep hacks.

Evidence: akkadotnet/Akka.Persistence.EventStore#76
Aaronontheweb added a commit to akkadotnet/akka.net that referenced this pull request Mar 17, 2026
* Add WaitForReadSideAsync() hook to Persistence TCK query specs

Add a protected virtual extensibility point to CurrentAllEventsSpec,
CurrentEventsByTagSpec, and CurrentEventsByPersistenceIdSpec that is
called after events are written and before queries are executed.

Backends with eventually-consistent read models (EventStore, Kafka,
etc.) can override this method to wait for their read side to catch up,
instead of being forced to override entire TCK test methods with
Thread.Sleep hacks.

Evidence: akkadotnet/Akka.Persistence.EventStore#76

* Make TCK query specs tolerant of eventually-consistent backends

Replace the virtual WaitForReadSideAsync() hook with inline polling
using AwaitConditionAsync. Each test now polls its respective Current*
query until the expected number of events are indexed before running
assertions. This is deterministic: synchronous backends pass on the
first poll, eventually-consistent backends converge.

Fixes the root cause of flaky TCK failures in backends like EventStore
where projections are eventually consistent. Previously, these backends
were forced to override entire test methods with Thread.Sleep hacks.

Evidence: akkadotnet/Akka.Persistence.EventStore#76
Aaronontheweb added a commit to akkadotnet/akka.net that referenced this pull request Apr 24, 2026
* Add WaitForReadSideAsync() hook to Persistence TCK query specs

Add a protected virtual extensibility point to CurrentAllEventsSpec,
CurrentEventsByTagSpec, and CurrentEventsByPersistenceIdSpec that is
called after events are written and before queries are executed.

Backends with eventually-consistent read models (EventStore, Kafka,
etc.) can override this method to wait for their read side to catch up,
instead of being forced to override entire TCK test methods with
Thread.Sleep hacks.

Evidence: akkadotnet/Akka.Persistence.EventStore#76

* Make TCK query specs tolerant of eventually-consistent backends

Replace the virtual WaitForReadSideAsync() hook with inline polling
using AwaitConditionAsync. Each test now polls its respective Current*
query until the expected number of events are indexed before running
assertions. This is deterministic: synchronous backends pass on the
first poll, eventually-consistent backends converge.

Fixes the root cause of flaky TCK failures in backends like EventStore
where projections are eventually consistent. Previously, these backends
were forced to override entire test methods with Thread.Sleep hacks.

Evidence: akkadotnet/Akka.Persistence.EventStore#76
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant