Support -parallel > 1 for real apply

[snaka]

Summary

-parallel currently only speeds up diff and apply -dry-run; a real apply always runs sequentially. This PR makes apply -parallel N (N > 1) run for real, through a three-phase plan → execute → summarize path that is deliberately more conservative than the sequential path.

-parallel 1 (the default) and -dry-run keep their existing code paths and byte-identical output — golden-log tests in this PR pin that.

⚠️ BREAKING CHANGE: duplicate rule names are rejected at load time

A config containing two rules with the same name no longer loads, for every subcommand, at any -parallel value. Configs that are valid today can start failing after this change.

It is deliberate rather than incidental. GetRuleByName resolves a name to the first match, so duplicates already caused two bugs: later rules were silently shadowed in sequential apply, and — worse — the same *Rule pointer was handed to several workers at once in the existing parallel diff / apply -dry-run, which is a data race. Making the new parallel apply safe means the duplicates have to be rejected rather than tolerated.

The error names the offending rule. Fixing a config means renaming the duplicate or regenerating with ecschedule dump.

How it works

sequenceDiagram
    autonumber
    participant M as runParallelApply
    participant W as workers<br/>(at most N in flight)
    participant AWS

    Note over M,AWS: Phase 1 — plan · read-only · N rules at a time
    M->>W: admit a rule (blocks while N are in flight)
    W->>AWS: DescribeTaskDefinition (memoized per task definition)
    W->>AWS: ListRules / ListTargetsByRule
    W-->>M: plan: differs? + rendered diff
    Note over M: BARRIER — every rule is planned before any write.<br/>One failure ⇒ report them all, exit 1, nothing written.

    Note over M,AWS: Phase 2 — execute · only rules that differ · N at a time
    M->>W: admit a changed rule
    rect rgba(127, 127, 127, 0.2)
    Note over W,AWS: ATOMIC per rule — a sibling's failure or the<br/>first Ctrl-C cannot interrupt this sequence<br/>(context.WithoutCancel + 2 min timeout)
    W->>AWS: PutRule
    W->>AWS: PutTargets
    W->>AWS: TagResource
    end
    W-->>M: outcome: applied / failed (never cancels siblings)
    Note over M: each rule's result is one indivisible log block,<br/>so results never interleave

    Note over M: Phase 3 — summary.<br/>prune runs only if every rule succeeded and nothing was interrupted.

Loading

Phase 1 — plan (read-only). Every rule is validated (env / tfstate / ssm placeholders, task definition existence) and diffed against the remote. No writes happen until every rule has passed. If any rule fails validation, all of them are reported and nothing is written — a bad config can no longer leave the account half-applied.

Phase 2 — execute. Only rules that actually differ are written, continue-on-error, with per-region clients (RetryMaxAttempts=10). Each rule's result — its diff plus the resulting YAML — is emitted as a single indivisible block, so results never interleave; the short applying rule "..." progress lines are separate and do appear between those blocks. A rule that has started writing always finishes its PutRule → PutTargets → TagResource sequence: the sequence runs under context.WithoutCancel plus a 2-minute timeout, so a sibling's failure — or the first Ctrl-C — can never interrupt a half-written rule, while a stalled AWS call still cannot hang the run forever.

Phase 3 — summary. Applied / failed / skipped counts, with the failures listed. runParallelApply returns nil only on full uninterrupted success, and that is what gates the -prune block, so pruning never runs on a partial result.

Behavior changes worth calling out

Besides the breaking change above:

• SIGINT/SIGTERM are now handled — all subcommands, any -parallel value. Previously nothing caught them, so a Ctrl-C during apply killed the process outright and could leave a rule with PutRule applied and PutTargets not. Now the first signal stops work at API-call boundaries, a rule that has already begun writing runs to completion, and the command exits 1 through the normal error path instead of dying to the signal (previously ~130). A second signal restores default fatal handling as an escape hatch.
• Tag-only failures are reported distinctly — apply -parallel > 1 only. When PutRule/PutTargets succeeded but TagResource did not, the failure prints a fully substituted aws events tag-resource repair command, because a re-run does not heal it. -parallel 1 keeps its existing error output byte for byte.

Caveat introduced by this PR

• The SDK retry budget is shared per region rather than per rule, so under sustained failures at -parallel > 1 some rules may surface a rate-limit-token error instead of the underlying one — two error texts for a single root cause. Documented in the README.

Two parallel runners, for now

executeJobsInParallelContinueOnError joins the existing executeJobsInParallel rather than replacing it, because the two differ in exactly one way that matters: the existing one shares an errgroup.WithContext, so the first error cancels the rest, while apply needs every rule attempted and the failures aggregated.

The split is not meant to be permanent, and the doc comment says so. executeJobsInParallelContinueOnError is the more general of the two — fail-fast can be layered on it by cancelling a derived context on the first error, while the reverse is not possible — so it is the consolidation target. The reason the remaining callers (diff and apply -dry-run) are not moved here is that it would change what they report, from only the first error to every rule's, and that behavior change does not belong in this PR.

Also fixed here: one DescribeTaskDefinition per rule

Planning validated the task definition once per rule, so a config where many rules share a task definition issued that many identical DescribeTaskDefinition calls. At -parallel 10 over 150 rules that is enough for ECS to answer ThrottlingException, and since planning is fail-closed the whole run aborted with a message blaming a task definition that exists. The last commit memoizes the result per region + task definition (so 150 calls become 1) and gives planning the same raised retry budget as the write phase. The dry-run and diff -validate parallel paths share the same memoization.

Testing

• Golden-log tests stub AWS at the HTTP layer and pin apply's exact log output and call sequence, so "the -parallel 1 path is unchanged" is enforced rather than asserted.
• Exercised against real AWS with a 150-rule config: fail-closed planning (confirmed remotely that nothing was written), a full apply followed by an idempotent re-run, the first Ctrl-C leaving no half-written rule, the tag-only failure path, and -prune deleting exactly the orphans.

The commits are ordered so each one builds and tests green on its own.

Pre-existing issues this PR does not address

Neither is introduced or changed by this PR, and neither is fixed here. They are listed only because applying many rules at once makes them easier to run into, and both look worth separate fixes.

• -prune orphan detection only searches the region of the ambient AWS configuration, so rules under a per-rule region override are outside its view.
• Changing a targetId leaves the old target attached remotely: the rule then fires both targets and subsequently diffs as a new rule.

[Copilot]

Pull request overview

This PR enables real parallel execution for apply -parallel N (N > 1) by introducing a conservative plan → execute → summarize workflow, while also hardening config loading and operational behavior (signals, retries, throttling). It fits into the codebase by expanding the existing parallel infrastructure (previously used for diff / apply -dry-run) to safely support parallel writes to EventBridge.

Changes:

• Add a new parallel apply engine (runParallelApply) that validates/diffs all rules before any write, then applies changed rules with aggregated failures and an end-of-run summary.
• Reject duplicate rule names at config load time (breaking change) to avoid shadowing and parallel data races.
• Improve resilience/operability: memoize DescribeTaskDefinition, add continue-on-error parallel runner, and add SIGINT/SIGTERM cancellation via signal.NotifyContext.

Show a summary per file

File	Description
rule.go	Refactors apply into plan/execute, adds task-definition validation memoization and tagging retry/error typing.
parallel.go	Adds executeJobsInParallelContinueOnError to run all jobs and aggregate outcomes without fail-fast cancellation.
parallel_apply.go	Implements the new three-phase parallel apply flow with per-rule logging and summary output.
cmd_apply.go	Removes the “dry-run only” parallel guard; routes real apply to runParallelApply when -parallel > 1.
cmd_diff.go	Shares task-definition validation across workers to reduce ECS API amplification.
config.go	Adds validateUniqueRuleNames and enforces it at config load time (breaking).
config_test.go	Adds coverage for duplicate rule name rejection.
cmd/ecschedule/main.go	Introduces SIGINT/SIGTERM handling using signal.NotifyContext, with a “second signal kills immediately” escape hatch.
README.md	Updates docs for real parallel apply behavior, failure semantics, and signal handling.
rule_apply_stub_test.go	Adds HTTP-layer AWS stubs and golden-log tests to pin output/call sequences.
parallel_continue_on_error_test.go	Tests the continue-on-error parallel runner behavior (success/failure/skip/panic).
parallel_apply_test.go	Tests parallel apply semantics (barrier planning, aggregation, memoization, interruption).

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 12/12 changed files
• Comments generated: 2
• Review effort level: Lite

[snaka]

Good catch — the wording was wrong, not the code, so I fixed the wording in 6291751.

You are right that the two log.Printf calls interleave. A real 150-rule run looks like this:

[ecschedule] applying rule "perf-test-rule-001"
[ecschedule] applying rule "perf-test-rule-004"
[ecschedule] ✅ rule "perf-test-rule-004" applied
💡 applied changes: ...
[ecschedule] applying rule "perf-test-rule-012"

What is indivisible is each rule's result: the ✅ rule … applied line, its diff and the resulting YAML go out in one log.Printf, so two results can never be spliced into each other. The claim of "one block per rule" overstated that, and the README and the PR body now say "result" explicitly.

I kept the progress line rather than folding it into the final block, for two reasons. A parallel apply is otherwise completely silent for its whole duration — the 150-rule run above takes ~25s — and this line is what tells you which rules are in flight. It also matches the sequential path, which prints the same applying the rule "…" before writing; dropping it only under -parallel > 1 would make the two paths diverge for no gain.

Happy to fold it in anyway if you would rather have exactly one line per rule.

[snaka]

Correct, and the exemption is in fact wider than this comment says. Rewritten in 6291751.

The old sentence asserted the general behavior and then immediately carved out the real one in a parenthetical, which reads as a contradiction. Worse, the parenthetical said "for parallel apply", but applyInternalWith also goes through execute() (rule.go), so the write sequence is shielded on the sequential path too — the exemption applies to any real apply, with or without -parallel.

The section now leads with the general behavior and states the exemption separately:

The first signal stops work at API-call boundaries and the command exits through the normal error path …

One part is deliberately exempt from that first signal: a rule whose write has already begun always completes its PutRule → PutTargets → TagResource sequence. That sequence is detached from the signal (it runs under context.WithoutCancel) and bounded only by the per-rule timeout, so no rule is ever left half-written. This holds for any real apply, with or without -parallel. What the first signal does stop in a parallel apply is admission: no further rules start, and the command waits for the in-flight ones.

The behavior was verified against a real 150-rule config: one SIGINT mid-run gave 40 applied, 0 failed, 110 not started, and every applied rule was written completely.