Skip to content

Work around SqlClient deadlock when a cancelled peek open enlists in the peek's TransactionScope - #1793

Open
SimonCropp wants to merge 1 commit into
Particular:masterfrom
SimonCropp:workaround-sqlclient-open-cancellation-deadlock
Open

SimonCropp wants to merge 1 commit into
Particular:masterfrom
SimonCropp:workaround-sqlclient-open-cancellation-deadlock

Conversation

@SimonCropp

Copy link
Copy Markdown
Contributor

This is a workaround, not a fix

The bug is in Microsoft.Data.SqlClient, filed as dotnet/SqlClient#4696. This PR only stops the transport from stepping into it. The change carries a TODO to undo it once that issue is fixed and the minimum SqlClient version this transport depends on carries the fix.

The problem

QueuePeeker.Peek opens its connection with the receive cancellation token, inside an ambient TransactionScope:

using (var scope = new TransactionScope(TransactionScopeOption.RequiresNew, ...))
using (var connection = await connectionFactory.OpenNewConnection(cancellationToken).ConfigureAwait(false))

MessageReceiver.StopReceive cancels that token, so every endpoint shutdown cancels an in-flight open. Against a Pooling=false connection string that intermittently deadlocks inside SqlClient:

thread A   SqlConnectionFactory.CreateReplaceConnectionContinuation keeps completing the open the
           caller has already abandoned. CompleteLogin -> EnlistNonNull registers the promotable
           single phase enlistment and then assigns EnlistedTransaction. The transaction has
           aborted by then, so add_TransactionCompleted fires the handler inline and
           DetachTransaction blocks on lock(connection).

thread B   the cancelled OpenAsync unwinds Peek, TransactionScope.Dispose rolls the transaction
           back, and SqlDelegatedTransaction.Rollback takes lock(connection) and then blocks on the
           parser lock thread A holds for the duration of the login.

Nothing breaks the cycle. MessageReceiver.StopReceive then awaits messageReceivingTask forever (there is no timeout on it), so the endpoint never shuts down.

Pooling=false is the precondition: SqlClient calls CompleteLogin(!ConnectionOptions.Pooling), and the auto-enlistment only runs inside the login, while the connection ctor holds the parser lock, when pooling is off. Test helpers that hand out non-pooled connection strings per test database hit this; pooled production connection strings should not.

This surfaced as CI builds hanging until the job timeout roughly once a week, with no test output and no exception. The hang dump is what identified it.

The change

Do not pass the token to the open, and return early when cancellation has already been requested. The open is then bounded by Connect Timeout instead of by the token.

Trade-off, stated plainly: shutdown now waits for an in-flight open to finish rather than cancelling it. Normally that is milliseconds. Against an unreachable server it is up to Connect Timeout (15s by default). That is a bounded wait in place of an unbounded hang, but it is a real change in shutdown behaviour and worth a second opinion. TryPeek and the trailing Task.Delay still observe the token, so the loop still exits promptly in the normal case.

This also affects the PostgreSql transport, since QueuePeeker.cs is compiled into both. I have not run the PostgreSql tests.

The test

When_peek_is_cancelled_during_connection_open drives the real QueuePeeker.Peek, cancelling the token partway through the open, and fails via a watchdog when a worker stops making progress. Two details matter and are commented in the test: it forces Pooling=false, and it spins on Stopwatch rather than using CancelAfter, because the OS timer (~15.6ms) is an order of magnitude coarser than a login and never lands inside the window.

Without the workaround With it
Repro test Deadlocked 2/2 runs — 22s / 7,303 peeks, and 1m22s / 15,676 peeks Passed 3/3, 2 minutes each
SqlServer.UnitTests 98 passed
SqlServer.IntegrationTests 19 passed

Verified against the pinned Microsoft.Data.SqlClient 6.1.6 and also 7.0.3.

It is marked [Explicit] on purpose and will not run in the normal suite: it burns up to two minutes when it passes, and when it reproduces it deliberately leaves deadlocked threads and a wedged SQL connection behind, so the test host is poisoned afterwards. Happy to drop it from this PR if you would rather not carry a stress test — the workaround stands on its own.

QueuePeeker.Peek opens its connection with the receive cancellation token inside
an ambient TransactionScope. MessageReceiver.StopReceive cancels that token, so
every endpoint shutdown cancels an in-flight open.

Against a Pooling=false connection string that deadlocks inside
Microsoft.Data.SqlClient. The client finishes the open the caller has already
abandoned and enlists it on one thread, while TransactionScope.Dispose rolls the
transaction back on another, and the two take the connection monitor and the
parser lock in opposite orders. The peek never completes, so StopReceive waits on
it forever and the endpoint never shuts down.

Stop passing the token to the open, and bail out early when cancellation has
already been requested. The open is then bounded by Connect Timeout instead of by
the token, while TryPeek and the trailing Task.Delay still observe it, so
shutdown stays prompt in the normal case.

This is a workaround, marked with a TODO to undo. The underlying bug is
dotnet/SqlClient#4696

Also adds an explicit stress test that reproduces the hang. It fails within a
minute without the workaround and passes with it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant