[Idea] Azure Storage Queues transport — Paramore.Brighter.MessagingGateway.AzureStorageQueues

[xbizzybone]

Summary

Brighter has first-class Azure support through Azure Service Bus, but there is currently no transport for Azure Storage Queues (the Azure.Storage.Queues SDK). We'd like to propose — and contribute and maintain — a new Paramore.Brighter.MessagingGateway.AzureStorageQueues package, plus an optional companion Paramore.Brighter.Outbox.AzureTableStorage.

We've been running a production command/queue pipeline on Azure Storage Queues + Table Storage for some time (visibility-delay scheduling, {queue}-poison dead-lettering, dequeue-count tracking), and we'd love to fold that experience back into Brighter rather than keep it in a private fork.

Why Storage Queues (and why it complements Service Bus)

Azure Storage Queues sit in a different niche from Service Bus and are a very common starting point on Azure:

• Ubiquitous & cheap — they ship with every Storage account, are billed per-transaction, and need no separate namespace/SKU. This is the default queue for a huge number of Azure workloads, especially Azure Functions (the Functions queue trigger is Storage Queues).
• Simple semantics that map cleanly onto Brighter — at-least-once delivery, per-message visibility timeout, native dequeue count, and the well-known <queue>-poison convention.
• Fills a real gap — teams that picked Storage Queues for cost/simplicity currently can't adopt Brighter's Command Processor + Service Activator without either migrating to Service Bus or writing a bespoke gateway. A first-class transport removes that friction and widens Brighter's Azure story from "Service Bus only" to "the whole Azure queue spectrum".

How it maps onto Brighter's model

The encouraging part is that Storage Queues fit Brighter's existing abstractions without bending them — and they fit the same shape as the AWS SQS gateway, which is the strongest precedent here because SQS, like Storage Queues, has no broker-native dead-letter and relies on visibility-timeout + a managed poison queue.

Proposed consumer mapping (IAmAMessageConsumerSync / IAmAMessageConsumerAsync):

Brighter operation	Azure Storage Queues API	Notes
Receive(timeout)	ReceiveMessages(maxMessages, visibilityTimeout)	maxMessages ← Subscription.BufferSize; stash PopReceipt + DequeueCount in MessageHeader.Bag (same idea as SQS ReceiptHandle / ASB LockToken).
Acknowledge(message)	DeleteMessage(messageId, popReceipt)	Delete on success.
Requeue(message, delay)	UpdateMessage(messageId, popReceipt, visibilityTimeout: delay)	Native visibility delay — identical strategy to the SQS gateway (ChangeMessageVisibility). Returns a refreshed PopReceipt.
Nack(message)	UpdateMessage(..., visibilityTimeout: 0)	Make immediately visible again.
Reject(message, reason)	produce to {queue}-poison (or invalid-message queue), then delete original	No native DLQ → follow Brighter-managed dead-letter, exactly like SqsMessageConsumer.

Producer mapping (IAmAMessageProducerSync / IAmAMessageProducerAsync):

Brighter operation	Azure Storage Queues API
Send(message)	SendMessage(body)
SendWithDelay(message, delay)	SendMessage(body, visibilityTimeout: delay) (initial-visibility delay, up to 7 days)

Dead-letter & poison handling would reuse the existing role interfaces rather than invent anything:

• Implement IUseBrighterDeadLetterSupport and IUseBrighterInvalidMessageSupport on the subscription (as SqsSubscription does).
• Route RejectionReason.DeliveryError → dead-letter channel and RejectionReason.Unacceptable → invalid-message channel.
• Default the dead-letter channel name to {queue}-poison — which is already the Azure Functions convention, so it will feel native to Azure users.
• Use the native DequeueCount to drive the "max receives → reject to poison" policy alongside Subscription.RequeueCount.

Proposed scope

Package 1 — Paramore.Brighter.MessagingGateway.AzureStorageQueues (its own assembly, per the "one transport, one assembly" rule):

• AzureStorageQueueMessageProducer / …ProducerFactory (IAmAMessageProducerSync + …Async)
• AzureStorageQueueMessageConsumer / …ConsumerFactory (IAmAMessageConsumerSync + …Async)
• AzureStorageQueueChannelFactory (IAmAChannelFactory)
• AzureStorageQueueSubscription / …Subscription<T> and AzureStorageQueuePublication / …Publication<T>
• Connection/auth providers mirroring the ASB ClientProvider family: connection string, DefaultAzureCredential, and Managed Identity via Azure.Identity.
• OnMissingChannel.Create/Validate/Assume support (create queue + {queue}-poison when asked).

Package 2 (optional companion) — Paramore.Brighter.Outbox.AzureTableStorage:

• Implements IAmAnOutboxSync<Message, …> + IAmAnOutboxAsync<Message, …>, following the non-relational outbox precedent (DynamoDB / MongoDB / Firestore) rather than the relational stores.
• Uses Table Storage entity-group transactions for the atomic "store state + enqueue intent" step, so teams whose source of truth already lives in Table Storage can use the Outbox pattern without standing up a relational DB.

Aligning with project conventions

We've read CONTRIBUTING.md, TESTING_GUIDE.md and the docs/transports/ pages, and intend to follow them to the letter:

• ADR first. Happy to open an ADR (using docs/adr/ template) covering the consumer semantics above — especially the requeue/visibility and poison-queue decisions — so the design is agreed before code. A draft ADR outline is appended below.
• Separate assembly, role-based interface naming, Nullable enabled, #region Licence headers, conventional commits, PRs targeting master.
• Central package management — add Azure.Storage.Queues (and Azure.Data.Tables for the outbox) to Directory.Packages.props; Azure.Identity is already there.
• TDD with xUnit, in-memory test doubles for unit tests, and integration tests against Azurite via a docker-compose-azurite.yaml (no cloud account needed in CI). Wired into the Liquid test-generator config if you'd like consistency with the other providers.
• Docs added under docs/transports/ and a README transport-list entry, plus a samples/ Task Queue example to match the RMQ/Kafka samples.

Open questions for the maintainers

A few decisions are genuinely yours to steer, and we'd like to align before writing code:

1. Requeue strategy — native UpdateMessage visibility (SQS-style, our preference) vs. re-send + delete (ASB-style). Any preference for consistency across the codebase?
2. Dead-letter model — confirm you'd want the Brighter-managed IUseBrighterDeadLetterSupport pattern with a {queue}-poison default, rather than anything bespoke.
3. SDK-version strategy — Azure.Storage.Queues v12 is stable, so we don't anticipate needing the side-by-side .V4-style split used for the AWS SDK. Agreed?
4. Naming — AzureStorageQueues (plural, matching the Azure.Storage.Queues SDK) vs. AzureStorageQueue. We lean plural; your call.
5. Outbox appetite — is the Table Storage outbox something you'd accept in-tree alongside the transport, or would you prefer to land the transport first and treat the outbox as a follow-up?
6. Ownership & maintenance — we're offering to author, test, and maintain this transport (CI, Azurite-based integration tests, release support) so it doesn't add unowned surface area for the core team. How would you like to structure that?

What we'd bring

• A production-validated mental model for the tricky bits: visibility-delay scheduling, {queue}-poison dead-lettering, and dequeue-count-driven retry policy.
• Willingness to do the full lifecycle — ADR → implementation → Azurite integration tests → docs → sample → ongoing maintenance.
• We're CLA-ready (Contributor Harmony Individual CLA via clahub) and can sign before the first PR.

Thanks for Brighter — it's a genuinely well-designed library, and we'd be glad to help round out its Azure coverage. Happy to jump on a call or hash out the ADR in whatever way is easiest for the team.

---

Appendix — Draft ADR outline (to open once the idea has traction)

File would land as docs/adr/NNNN-azure-storage-queues-transport.md using the repo's ADR template.

• Status: Proposed
• Context:
  • Brighter supports Azure Service Bus but not Azure Storage Queues; Storage Queues are the default low-cost queue on Azure and the backbone of the Functions queue trigger.
  • Storage Queues have no native dead-letter; they expose per-message visibility timeout and dequeue count — the same constraints the AWS SQS gateway already handles.
• Decision:
  • Add Paramore.Brighter.MessagingGateway.AzureStorageQueues implementing the standard producer/consumer/channel-factory contracts.
  • Map Acknowledge → DeleteMessage, Requeue/Nack → UpdateMessage(visibilityTimeout), SendWithDelay → SendMessage(visibilityTimeout).
  • Model dead-lettering with IUseBrighterDeadLetterSupport / IUseBrighterInvalidMessageSupport, defaulting the poison channel to {queue}-poison; drive the policy from native DequeueCount + Subscription.RequeueCount.
  • Stash PopReceipt + DequeueCount in MessageHeader.Bag.
  • Auth via connection string / DefaultAzureCredential / Managed Identity, mirroring the ASB client-provider family.
• Consequences:
  • Brighter gains the full Azure queue spectrum; no breaking changes to core.
  • New external dependency Azure.Storage.Queues (centrally managed); integration tests rely on Azurite, so CI needs no Azure account.
  • Optional follow-up: Paramore.Brighter.Outbox.AzureTableStorage on the non-relational outbox precedent (DynamoDB/MongoDB/Firestore).