decrypt the Slack token when dispatching metric alert notifications - #8305
Merged
Conversation
Collaborator
Contributor
|
🐋 This PR was built and pushed to the following Docker images: Targets: Platforms: Image Tags: |
n1ru4l
reviewed
Aug 3, 2026
n1ru4l
approved these changes
Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fixes metric alert Slack notifications failing with
invalid_auth.organizations.slack_tokenis written encrypted by the API, but the workflows service read the column raw and handed the ciphertext to Slack as a bearer token. Every org that connected Slack after encryption was introduced is affected.To fix it, the encryptor moves out of the API's
CryptoProviderinto@hive/service-commonso services without graphql-modules can decrypt, with a frozen ciphertext vector pinning the wire format. The workflows service gains anENCRYPTION_SECRETvariable (Pulumi, compose, env template), kept optional so an existing self-hosted deployment keeps booting on upgrade: without it Slack notifications are skipped, rather than the whole service, including all transactional email and crons, failing to start. The dispatcher then decrypts the token before constructing the Slack client, usingpossiblyRawso legacy plaintext tokens keep working, and returnsfailed-configinstead of throwing when the secret is missing or wrong, so an unfixable error cannot burn all 25 graphile-worker retries.While tracing this, two paths turned out to record
outcome = 'sent'while delivering nothing: a channel missing its settings, and a webhook whose retry budget was already spent. Both now reportskipped-configandgave-up, the dedupe row is only written on a real delivery, and the Grafana panel documents the new outcomes. This is why the dashboard showed no failures for the misconfigured cases.Note that the fix is not retroactive: already-failed jobs sit at
attempts = 25and are never retried, so alerts from the broken window are lost and notifications resume on the next state transition. Self-hosters running custom manifests need to setENCRYPTION_SECRETon the workflows service to the same value as the API.