Schedule deadline reminder command with EventBridge ECS task

[alexeygrigorev]

Context

Issue #180 covers deadline reminder email behavior. We need a deployed scheduling mechanism that runs CMP reminder logic without tying it to web requests.

We do not need Lambda for this. EventBridge should start a one-off ECS task using the CMP image.

Current environment note: CMP deploys to dev via CI/CD, and Datamailer currently has only a sandbox deployment. There is no Datamailer production environment for this work.

Desired runtime

EventBridge schedule -> ECS RunTask with CMP image -> uv run python manage.py send_deadline_reminders -> Datamailer enqueue/list-send API -> exit

The scheduled command should be quick. It should compute eligible reminder windows, create/trigger Datamailer work, and exit. Datamailer handles slow per-recipient sending asynchronously.

Desired behavior

• Add send_deadline_reminders management command if not already implemented by Add deadline reminder emails for homework, projects, and peer reviews #180.
• Add an --enqueue-only or equivalent mode if needed so the command never waits for delivery.
• Configure infra/deployment docs for EventBridge scheduled ECS task.
• Ensure the command is safe to run repeatedly.
• Use stable idempotency/send keys for reminder windows.

Acceptance criteria

• Local command can be run manually with uv run python manage.py send_deadline_reminders.
• Dev schedule is documented/configured as EventBridge -> ECS task, with no Lambda requirement.
• The command exits quickly after triggering Datamailer work.
• Duplicate scheduled runs do not send duplicate reminders.
• Failures are logged and observable.

Related: #180

Current blocker

Repo-side implementation and documentation are in place. The remaining step is external AWS IAM/scheduler state: grant the CI deploy user the documented permissions, or provide an existing Scheduler role ARN, then rerun the Configure Deadline Reminders workflow and verify the EventBridge schedule exists.

[alexeygrigorev]

Code/docs are in place and deployed to dev in 426568c: deploy/schedule_deadline_reminders.sh configures EventBridge Scheduler -> ECS RunTask for uv run python manage.py send_deadline_reminders, and docs/README describe the flow. I am leaving this open because creating/verifying the actual AWS schedule still requires AWS credentials and SCHEDULER_ROLE_ARN; local AWS access currently fails with AWS Gate HTTP 403.

[alexeygrigorev]

Added a CI/CD path for configuring the schedule in c817cd3: .github/workflows/configure-deadline-reminders.yaml exposes a manual "Configure Deadline Reminders" workflow that runs deploy/schedule_deadline_reminders.sh with the same AWS credentials as dev deploy. It takes environment, scheduler_role_arn, schedule_expression, and optional schedule_name inputs. README and docs/datamailer-protocol.md now document both local and workflow-dispatch setup.\n\nLeaving this open until the actual EventBridge schedule is created/verified. The missing input is the Scheduler role ARN trusted by scheduler.amazonaws.com; local AWS access still fails through AWS Gate, so I cannot discover or create that role from this machine.

[alexeygrigorev]

Progress update:

Implemented additional scheduler setup automation in 773e85b and follow-up diagnostics in @6aa1e28.

What is now in the repo:

• deploy/schedule_deadline_reminders.sh no longer requires SCHEDULER_ROLE_ARN up front.
• If no ARN is provided, it attempts to create/update course-management--deadline-reminders-scheduler, trust scheduler.amazonaws.com, and attach a least-scoped inline policy for ecs:RunTask on the current task definition plus iam:PassRole for the task roles.
• The Configure Deadline Reminders workflow now has scheduler_role_arn as optional.
• README and docs/datamailer-protocol.md document both modes and required IAM permissions.
• Avoided ecs:DescribeClusters because the CI deploy user lacks that action; the script now uses clusterArn from describe-services.

Verification:

• bash -n deploy/schedule_deadline_reminders.sh passed.
• uv run python manage.py test courses.tests.test_deadline_reminders -> 4 tests OK.
• Dev Deploy for 773e85b passed.

Actual schedule creation attempt:

• Workflow run 27824083458 failed because CI user lacked ecs:DescribeClusters. Fixed by 0249f1d.
• Workflow run 27824158772 then failed at IAM role creation: course-management-ci-cd-deploy-user is not authorized for iam:CreateRole on course-management-dev-deadline-reminders-scheduler.

Remaining blocker to fully close this issue:

• Either grant the CI deploy user IAM permissions for the scheduler role (at least iam:GetRole, iam:CreateRole, iam:PutRolePolicy, and likely iam:PassRole for that role path), then rerun Configure Deadline Reminders without scheduler_role_arn; or
• Create/provide an existing EventBridge Scheduler role trusted by scheduler.amazonaws.com and rerun Configure Deadline Reminders with scheduler_role_arn set.

Leaving open because the real EventBridge schedule is still not created/verified.

[alexeygrigorev]

Final code-side update:

• 773e85b made the Scheduler role ARN optional and added automatic role/policy creation.
• 0249f1d removed the unnecessary ecs:DescribeClusters call after the configure workflow showed the CI deploy user lacks that action.
• 6aa1e28 added clearer failure messaging and docs for the remaining IAM permissions.

Verification:

• bash -n deploy/schedule_deadline_reminders.sh passed.
• uv run python manage.py test courses.tests.test_deadline_reminders -> 4 tests OK.
• Deploy Dev for 6aa1e28 passed.
• Dev health now reports 20260619-120107-6aa1e28.

Schedule creation attempts:

• 27824083458 failed on ecs:DescribeClusters; fixed by 0249f1d.
• 27824158772 failed on iam:CreateRole for course-management-dev-deadline-reminders-scheduler.

At this point I do not see more repo-side work to do for #188. The remaining requirement is external AWS IAM state: grant the GitHub deploy user permission to manage the scheduler role, or provide an existing Scheduler role ARN and rerun the Configure Deadline Reminders workflow.

[alexeygrigorev]

Added the missing IAM runbook in 7b3cea5.

What changed:

• New docs/deadline-reminder-scheduler-iam.md with the exact CI deploy-user permissions for Configure Deadline Reminders.
• Same doc includes the alternative path where a human pre-creates the EventBridge Scheduler role and passes scheduler_role_arn to the workflow.
• README and docs/datamailer-protocol.md now link to this runbook.
• Protocol wording now says deployed CMP environments instead of implying Datamailer production; Datamailer is still sandbox-only.

Verification:

• git diff --check passed.
• bash -n deploy/schedule_deadline_reminders.sh passed.
• Deploy Dev for 7b3cea5 passed.
• Dev health now reports 20260619-121341-7b3cea5.

Remaining blocker is still external AWS IAM state, not repo code: grant the CI deploy user the documented IAM permissions, or provide an existing Scheduler role ARN and rerun Configure Deadline Reminders. Then verify the EventBridge schedule exists and targets the ECS RunTask command for python manage.py send_deadline_reminders.