Skip to content

decrypt the Slack token when dispatching metric alert notifications - #8305

Merged
jonathanawesome merged 7 commits into
mainfrom
fix-metric-alert-slack-token-decryption
Aug 3, 2026
Merged

decrypt the Slack token when dispatching metric alert notifications#8305
jonathanawesome merged 7 commits into
mainfrom
fix-metric-alert-slack-token-decryption

Conversation

@jonathanawesome

Copy link
Copy Markdown
Member

This PR fixes metric alert Slack notifications failing with invalid_auth. organizations.slack_token is written encrypted by the API, but the workflows service read the column raw and handed the ciphertext to Slack as a bearer token. Every org that connected Slack after encryption was introduced is affected.

To fix it, the encryptor moves out of the API's CryptoProvider into @hive/service-common so services without graphql-modules can decrypt, with a frozen ciphertext vector pinning the wire format. The workflows service gains an ENCRYPTION_SECRET variable (Pulumi, compose, env template), kept optional so an existing self-hosted deployment keeps booting on upgrade: without it Slack notifications are skipped, rather than the whole service, including all transactional email and crons, failing to start. The dispatcher then decrypts the token before constructing the Slack client, using possiblyRaw so legacy plaintext tokens keep working, and returns failed-config instead of throwing when the secret is missing or wrong, so an unfixable error cannot burn all 25 graphile-worker retries.

While tracing this, two paths turned out to record outcome = 'sent' while delivering nothing: a channel missing its settings, and a webhook whose retry budget was already spent. Both now report skipped-config and gave-up, the dedupe row is only written on a real delivery, and the Grafana panel documents the new outcomes. This is why the dashboard showed no failures for the misconfigured cases.

Note that the fix is not retroactive: already-failed jobs sit at attempts = 25 and are never retried, so alerts from the broken window are lost and notifications resume on the next state transition. Self-hosters running custom manifests need to set ENCRYPTION_SECRET on the workflows service to the same value as the API.

@theguild-bot

theguild-bot commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

🚀 Snapshot Release (alpha)

The latest changes of this PR are available as alpha on npm (based on the declared changesets):

Package Version Info
hive 11.9.1-alpha-20260803174319-46b0c5745755a01098fbdaff8ecd8196f73ae08b npm ↗︎ unpkg ↗︎

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

🐋 This PR was built and pushed to the following Docker images:

Targets: build

Platforms: linux/amd64

Image Tags: 11.9.1-alpha-46b0c57, 46b0c57, 46b0c5745755a01098fbdaff8ecd8196f73ae08b

Comment thread packages/services/service-common/src/crypto.ts Outdated
@jonathanawesome
jonathanawesome merged commit dee12d9 into main Aug 3, 2026
48 of 49 checks passed
@jonathanawesome
jonathanawesome deleted the fix-metric-alert-slack-token-decryption branch August 3, 2026 18:13
@theguild-bot
theguild-bot temporarily deployed to staging August 3, 2026 18:19 Inactive
@theguild-bot
theguild-bot temporarily deployed to production August 3, 2026 18:38 Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants