Move alerting to Python by ericboucher · Pull Request #1931 · WFP-VAM/prism-app

ericboucher · 2026-06-05T16:00:22Z

Description

Removing node backend and move our alerting mechanisms to the frontend.

Workers: WFP data fetch, Playwright map capture, Jinja2 email templates, SMTP/Ethereal delivery via export_map_worker
CLI: python -m prism_app.workers.alert_runner (threshold, aa-flood, aa-storm, smoke), api crons
Tests: unit tests, CI smoke/contract jobs, Ethereal e2e harness (make api-alert-email-e2e)
Dev tooling: make send-test-email (Ethereal preview with --test-email / is_test bypass when trigger is not exceeded)

How to test the feature:

cd api && make api, make db-seed
make send-test-email (flood email shown below, open Ethereal preview URL email in logs)
make send-test-email WORKER=aa-storm (storm email shown below, open Ethereal preview URL email in logs)
make api-alert-email-e2e(slow; flood/storm/threshold layout parity)

Checklist - did you ...

Test your changes with

REACT_APP_COUNTRY=rbd yarn start
REACT_APP_COUNTRY=cambodia yarn start
REACT_APP_COUNTRY=mozambique yarn start
Add / update necessary tests?
Add / update outdated documentation?

Post-deploy checklist

crontab -e updates on EC2 instance
Ensure cron scripts executable: chmod +x api/crons/cron_*.sh api/crons/_compose_run.sh
Remove old alerting-node container if running
Manual preflight check: cd ~/prism-app/api source set_envs.sh docker compose run --rm --no-deps export_map_worker \ python -c "from prism_app.alert_workers.smtp_mailer import require_smtp_configured; require_smtp_configured(); print('SMTP OK')"
Can optionally send a test email using both aa-flood and aa-storm
Check that all three crons run at least once post-migration

Future considerations for improvement

Add delivery audit table (right now email tracking done through logs and looking at SES)
last_triggered or last_states getting updated are not proof of mail sent successfully
Overlapping cron instances can hypothetically double send, as they don't use row-locking
Storm worker marks all latest reports processed even if one storm fails mid-run

Screenshot/video of feature:

Flood alert email (ethereal)

Storm alert email (ethereal)

Threshold alert email (ethereal)

… into cam/schedule-batch-map

Reconcile scheduled batch maps with master's WMS cadence cron and API schedules; keep Python alert workers on prism_app.workers.alert_runner.

github-actions · 2026-06-05T16:02:31Z

Visit the preview URL for this PR (updated for commit 7a985b5):

https://staging-prism-frontend--1931-g7yvfz03.web.app

_{(expires Sat, 18 Jul 2026 22:13:54 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

_{Sign: 4de97634797dc19a4f4938b370446c2b3ddc1c37}

Poetry lock was stale after alert worker deps landed, breaking all API CI jobs. Regenerate lock, fix isort on alert modules, bump pip floor to >=26.1.2 for pip-audit (CVE-2026-8643).

Add Cursor rule and prepare-commit-msg hook to strip Cursor trailers. Run ./scripts/install-githooks.sh once per clone.

Copilot

Pull request overview

This PR is a WIP migration of the alerting system away from the Node-based alerting/ service into Python under the existing api/ package, while also adding a small frontend UX enhancement for copying batch map export settings.

Changes:

Port threshold + anticipatory-action (storm/flood) alert workers to Python (api/prism_app/alert_workers/*) and wire them into CI + cron scripts.
Remove the Node alerting/ service, its tests, and its dedicated GitHub Actions workflow.
Add a frontend “copy batch map settings” action that copies a readable /export URL to clipboard.

Reviewed changes

Copilot reviewed 110 out of 114 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
scripts/install-githooks.sh	Adds helper to point git hooks to `.githooks/`.
scripts/ec2-security-patch.sh	Stops updating yarn lockfiles for the removed `alerting/` package.
frontend/src/components/NavBar/PrintImage/printConfig.tsx	Adds a copy-to-clipboard button for batch map settings.
frontend/src/components/NavBar/PrintImage/printConfig.context.ts	Exposes `copyBatchMapUrls` in the print config context.
frontend/src/components/NavBar/PrintImage/image.tsx	Implements `copyBatchMapUrls` and clipboard write behavior.
frontend/src/components/NavBar/PrintImage/batchExportUrls.ts	Adds helper to format export URLs for clipboard readability.
frontend/src/components/NavBar/PrintImage/batchExportUrls.test.ts	Unit test for clipboard URL formatting.
api/scripts/alert_email_ethereal_e2e_docker.sh	Adds Docker-driven E2E runner for Ethereal email previews.
api/requirements-export-map-worker.txt	Expands worker image requirements to support alert workers + templating.
api/README.md	Updates documentation to reflect Python alerting and new run paths.
api/pyproject.toml	Adds Python deps for the new alert worker stack (croniter, jinja2).
api/prism_app/workers/alert_runner.py	Adds CLI entrypoint to run threshold + AA workers and CI smoke.
api/prism_app/workers/init.py	Updates workers package docstring to include alert crons.
api/prism_app/tests/fixtures/moz_export.py	Adds helper for newline-separated schedule URL text fixtures.
api/prism_app/database/init.py	Exposes `MapExportSchedule` from the database package.
api/prism_app/ci/alert_db_smoke.py	Adds Python-based alerts DB smoke checks (SELECTs).
api/prism_app/ci/alert_db_contract.py	Adds Python-based alerts schema contract checks.
api/prism_app/ci/init.py	Adds CI helper package marker.
api/prism_app/alert_workers/init.py	Adds alert workers package marker.
api/prism_app/alert_workers/settings.py	Adds env/settings helpers for alert workers.
api/prism_app/alert_workers/db.py	Adds SQLAlchemy-based alerts DB access helpers.
api/prism_app/alert_workers/wcs_url.py	Adds WCS URL builders used by threshold worker.
api/prism_app/alert_workers/bbox_utils.py	Adds bbox helpers aligned with frontend/common behavior.
api/prism_app/alert_workers/layer_days.py	Adds WMS/WCS time-availability fetchers for alerts.
api/prism_app/alert_workers/mail_render.py	Adds Jinja2 rendering for storm/flood/threshold emails.
api/prism_app/alert_workers/smtp_mailer.py	Adds SMTP sender with Ethereal integration and CID images.
api/prism_app/alert_workers/browser_shot.py	Adds Playwright screenshot capture for AA email maps.
api/prism_app/alert_workers/coastal_districts.py	Ports coastal district list for storm alert filtering.
api/prism_app/alert_workers/threshold_worker.py	Ports threshold alert worker logic from Node to Python.
api/prism_app/alert_workers/aa_storm.py	Ports AA storm worker logic from Node to Python.
api/prism_app/alert_workers/aa_flood.py	Ports AA flood worker logic from Node to Python.
api/prism_app/alert_workers/templates/threshold_alert.txt.j2	Adds threshold email text template (Jinja2).
api/prism_app/alert_workers/templates/threshold_alert.html.j2	Adds threshold email HTML template (Jinja2).
api/prism_app/alert_workers/templates/storm_alert.txt.j2	Adds storm email text template (Jinja2).
api/prism_app/alert_workers/templates/storm_alert.html.j2	Adds storm email HTML template (Jinja2).
api/prism_app/alert_workers/templates/flood_alert.txt.j2	Adds flood email text template (Jinja2).
api/prism_app/alert_workers/templates/flood_alert.html.j2	Adds flood email HTML template (Jinja2).
api/prism_app/alert_workers/assets/mapIcon.png	Adds icon asset for AA emails.
api/prism_app/alert_workers/assets/arrowForwardIcon.png	Adds icon asset for AA/threshold emails.
api/poetry.lock	Locks added Python dependencies.
api/Makefile	Adds `api-alert-email-e2e` convenience target.
api/Dockerfile.export-map-worker	Bumps pip tooling version for worker image.
api/Dockerfile	Bumps pip tooling version for API image.
api/docker-compose.yml	Adds shm sizing and env wiring for alert worker execution; adds local alerts PostGIS service.
api/crons/_compose_run.sh	Adds shared compose runner for one-shot cron commands with logging.
api/crons/cron_scheduled_public_maps.sh	Refactors scheduled public maps cron to use `_compose_run.sh`.
api/crons/cron_alert_run.sh	Adds threshold alerts cron wrapper.
api/crons/cron_aa_storm_alert_run.sh	Adds AA storm cron wrapper.
api/crons/cron_aa_flood_alert_run.sh	Adds AA flood cron wrapper.
api/.gitignore	Ignores local alerts DB docker volume directory.
api/.env.example	Adds alert SMTP env placeholders.
.github/workflows/api.yml	Updates CI to run Python alert DB contract/smoke + adds unit test job for alert workers.
.github/workflows/alerting.yml	Removes Node alerting workflow.
.githooks/prepare-commit-msg	Adds hook to strip Cursor co-author trailers.
.dockerignore	Removes ignore patterns for deleted `alerting/` paths.
.cursor/rules/git-commits-no-cursor.mdc	Adds Cursor rule doc for commit attribution.
alerting/tsconfig.json	Deletes Node alerting TS config (alerting migration).
alerting/src/utils/raster-utils.ts	Deletes Node alerting utility.
alerting/src/utils/jest.config.js	Deletes Node alerting Jest config.
alerting/src/utils/email.ts	Deletes Node SMTP/email sender implementation.
alerting/src/utils/email.test.ts	Deletes Node SMTP/email sender tests.
alerting/src/utils/date.ts	Deletes Node date helper.
alerting/src/utils/date.test.ts	Deletes Node date helper tests.
alerting/src/utils/capture-utils.ts	Deletes Node Puppeteer screenshot code.
alerting/src/utils/capture-utils.test.ts	Deletes Node screenshot tests.
alerting/src/utils/analysis-utils.ts	Deletes Node threshold analysis code.
alerting/src/types/storm-reports.ts	Deletes Node alerting types.
alerting/src/types/storm-email.ts	Deletes Node alerting types.
alerting/src/types/flood-email.ts	Deletes Node alerting types.
alerting/src/types/anticipatory-action-alerts.ts	Deletes Node alerting types.
alerting/src/types/alert.ts	Deletes Node alerting types.
alerting/src/templates/storm-alert.ejs	Deletes Node storm email templates (EJS).
alerting/src/templates/flood-alert.ejs	Deletes Node flood email templates (EJS).
alerting/src/images/index.ts	Deletes Node image helper.
alerting/src/db/row-mappers.ts	Deletes Node DB row mappers.
alerting/src/db/row-mappers.test.ts	Deletes Node DB row mappers tests.
alerting/src/db/pool.ts	Deletes Node Postgres pool management.
alerting/src/db/alert-queries.ts	Deletes Node threshold alert DB queries.
alerting/src/db/aa-queries.ts	Deletes Node AA DB queries.
alerting/src/constants.ts	Deletes Node API URL constants.
alerting/src/ci/smoke-alerts-db-pool.ts	Deletes Node DB smoke script.
alerting/src/ci/smoke-alerting-workers.ts	Deletes Node worker smoke script.
alerting/src/ci/check-alerts-db-contract.ts	Deletes Node schema contract script.
alerting/src/alert-worker.ts	Deletes Node threshold worker.
alerting/src/alert-worker.test.ts	Deletes Node threshold worker tests.
alerting/src/aa-storm-alert/worker.ts	Deletes Node AA storm worker.
alerting/src/aa-storm-alert/worker.test.ts	Deletes Node AA storm worker tests.
alerting/src/aa-storm-alert/test-utils.ts	Deletes Node AA storm test utils.
alerting/src/aa-storm-alert/districs.ts	Deletes Node AA storm districts list.
alerting/src/aa-storm-alert/alert.ts	Deletes Node AA storm logic module.
alerting/src/aa-storm-alert/alert.test.ts	Deletes Node AA storm tests.
alerting/src/aa-storm-alert/snapshots/alert.test.ts.snap	Deletes Node snapshot file.
alerting/src/aa-storm-alert-runner.ts	Deletes Node storm runner.
alerting/src/aa-flood-alert/worker.ts	Deletes Node AA flood worker.
alerting/src/aa-flood-alert/alert.ts	Deletes Node AA flood logic module.
alerting/src/aa-flood-alert-runner.ts	Deletes Node flood runner.
alerting/src/aa-common/runner.ts	Deletes Node shared AA runner.
alerting/src/aa-common/runner.test.ts	Deletes Node AA runner tests.
alerting/README.md	Deletes Node alerting README (moved to API docs).
alerting/package.json	Deletes Node alerting package manifest and scripts.
alerting/Dockerfile	Deletes Node alerting Dockerfile.
alerting/docker-compose.yml	Deletes Node alerting compose stack.
alerting/crons/cron_alert_run.sh	Deletes Node alert cron script.
alerting/crons/cron_aa_storm_alert_run.sh	Deletes Node storm cron script.
alerting/crons/cron_aa_flood_alert_run.sh	Deletes Node flood cron script.
alerting/.prettierrc	Deletes Node formatting config.
alerting/.gitignore	Deletes Node ignore file.
alerting/.eslintrc	Deletes Node eslint config.
alerting/.eslintignore	Deletes Node eslint ignore.
alerting/.env.example	Deletes Node alerting env example.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

ericboucher · 2026-06-11T17:18:31Z

@copilot fix failing linting and CI

Copilot · 2026-06-11T17:25:28Z

@copilot fix failing linting and CI

Fixed in ccfc2b7. The current API failures were caused by a syntax regression in api/prism_app/alert_workers/threshold_worker.py; I restored the missing url_params = format_prism_url(...) assignment and re-ran the corresponding local API formatting checks and test_alert_workers_no_db.

…e-alerting

c-mac and others added 18 commits May 13, 2026 09:36

new schedule table and model; add FK to export

507b6a5

schedule job service

1970e69

worker polls for scheduled jobs

10c801d

schedule admin view

22d40e3

scheduler tests

64e7e9f

lint

08dc515

frontend: copy batch map url

6462d31

batch map url instead of json

6991c83

worker on compose network

6a022b8

Merge branch 'master' into cam/schedule-batch-map

805f202

fix double copy

9c77fb5

Merge branch 'cam/schedule-batch-map' of github.com:WFP-VAM/prism-app…

91b99bf

… into cam/schedule-batch-map

fix lint

d78b746

disable flag

8c8d973

fix fe test

f7d2ba3

Move alerting to python

325d797

improve threshold email styling

729c632

Merge branch 'master' into move-alerting

bcc1466

Reconcile scheduled batch maps with master's WMS cadence cron and API schedules; keep Python alert workers on prism_app.workers.alert_runner.

ericboucher added 3 commits June 5, 2026 19:16

Delete alerting node folder and mentions

f12671a

fix(api): regenerate poetry.lock and CI lint deps

1377a37

Poetry lock was stale after alert worker deps landed, breaking all API CI jobs. Regenerate lock, fix isort on alert modules, bump pip floor to >=26.1.2 for pip-audit (CVE-2026-8643).

chore: block Cursor co-author on git commits

122bdbe

Add Cursor rule and prepare-commit-msg hook to strip Cursor trailers. Run ./scripts/install-githooks.sh once per clone.

ericboucher force-pushed the move-alerting branch from b1141bf to 122bdbe Compare June 5, 2026 17:59

ericboucher added 2 commits June 6, 2026 00:08

Simplify

936f3a1

Merge branch 'master' into move-alerting

b5135dd

ericboucher requested a review from Copilot June 9, 2026 19:21

Copilot started reviewing on behalf of ericboucher June 9, 2026 19:22 View session

Copilot AI reviewed Jun 9, 2026

View reviewed changes

Comment thread api/prism_app/alert_workers/threshold_worker.py

Comment thread api/prism_app/alert_workers/smtp_mailer.py Outdated

ericboucher and others added 2 commits June 11, 2026 19:14

Merge branch 'master' into move-alerting

e2437d8

Potential fix for pull request finding

171a9e2

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

d645532

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

ericboucher assigned c-mac Jun 11, 2026

Copilot started work on behalf of ericboucher June 11, 2026 17:18 View session

fix threshold worker syntax regression

ccfc2b7

Copilot finished work on behalf of ericboucher June 11, 2026 17:25

ericboucher and others added 16 commits June 12, 2026 10:31

Merge branch 'master' into move-alerting

fce1aa3

Merge branch 'master' into move-alerting

0cc61f2

Merge branch 'master' into move-alerting

46c8191

Merge branch 'move-alerting' of github.com:WFP-VAM/prism-app into mov…

709e2e2

…e-alerting

remove orphaned alerting files

b736099

remove redeundant alerting-db compose service

374f98e

port over old ssh setting from alerting cron connection

f20ea8c

port over old alerting tests

920fb6c

db connection consistency

a94ee3e

send test flood emails

82c8b0a

update lock file

40d507e

Merge branch 'master' into move-alerting

f24c915

fix lock

0a36d83

fix snapshot

0aa9b5c

harden worker error handling, fail fast in prod if no smtp creds

a7f68f3

no ethereal in prod

b7d8afb

c-mac changed the title ~~WIP - Move alerting to Python~~ Move alerting to Python Jun 18, 2026

Merge branch 'master' into move-alerting

6390450

c-mac marked this pull request as ready for review June 18, 2026 15:18

c-mac requested a review from wadhwamatic as a code owner June 18, 2026 15:18

Merge branch 'master' into move-alerting

7a985b5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move alerting to Python#1931

Move alerting to Python#1931
ericboucher wants to merge 45 commits into
masterfrom
move-alerting

ericboucher commented Jun 5, 2026 •

edited by c-mac

Loading

Uh oh!

github-actions Bot commented Jun 5, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

ericboucher commented Jun 11, 2026

Uh oh!

Copilot AI commented Jun 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ericboucher commented Jun 5, 2026 • edited by c-mac Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How to test the feature:

Checklist - did you ...

Post-deploy checklist

Future considerations for improvement

Screenshot/video of feature:

Uh oh!

github-actions Bot commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

ericboucher commented Jun 11, 2026

Uh oh!

Copilot AI commented Jun 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ericboucher commented Jun 5, 2026 •

edited by c-mac

Loading

github-actions Bot commented Jun 5, 2026 •

edited

Loading