Add "Paused" state for dag steps

[wzhpor]

Description

Add a non-terminal task instance state paused that lets an operator hold a
specific task instance before it runs, and later resume it without failing it,
without clearing it, and without occupying a worker or pool slot.

Behaviour:

• scheduled / queued -> paused via UI, CLI or REST.
• paused -> scheduled on resume; try_number and XComs preserved. The
  scheduler picks it up on the next loop, same as a cleared task.
• Downstream tasks stay blocked while the task is paused; the DAG run stays
  running.
• paused does not consume worker slots or pool slots (like deferred).
• Optional pause_reason, paused_by, and resume_at fields for audit and
  scheduled auto-resume.

Pausing a running task is out of scope for the initial proposal. If an
operator realises too late, they can pause downstream tasks to hold the
workflow. Support for pausing running tasks (kill vs. let-finish-and-discard)
can be considered as a follow-up.

Use case / motivation

Enterprise schedulers (UC4/Automic, Control-M, TWS) let an operator pause an
individual step when a downstream dependency is temporarily broken (DB down,
partner SFTP unavailable, maintenance window, ad-hoc manual check). Once the
issue is fixed out-of-band, the operator resumes the step and the workflow
continues without re-running upstream work.

In Airflow today the only options are:

1. Let the task fail and clear it later — re-runs work, burns retries, fires
   failure alerts, breaks deadline/SLA alerts.
2. Keep it running and burn a worker slot for hours while ops fixes the
   underlying issue.
3. Mark it success manually — loses the guarantee that the step actually ran.

Existing mechanisms don't cover the ad-hoc operational case:

• dags pause only stops future DAG runs, not an individual task instance.
• deferred requires operator-level code changes and a trigger; it can't be
  applied ad-hoc by an operator.
• HITL / awaiting_input must be declared at DAG-authoring time; it can't be
  applied to an arbitrary task on demand.
• Zero-slot pools block new tasks from starting but don't pause ones already
  scheduled/queued against a different pool.

The goal is operational transparency: a distinct, non-terminal state with
its own audit trail (paused_by, pause_reason) instead of overloading
failed + clear for planned operational holds.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[jroachgolf84]

@amoghrajesh - I'd be curious to get your thoughts on this one?

[jroachgolf84]

I think there are some questions to be answered:

• What happens if you pause mid-run. Should the operation going on behind the scenes continue to execute? Should it fail?
• When you un-pause, where does it pick up? Does it start over?

Resumable Operators are definitely something to think about here.

[wzhpor]

I ammended my original propsal with narrow the scope:

1. Pausing mid-run: I'd drop that from the initial proposal. Pause would only apply to tasks that are not yet running (scheduled / queued, and possibly deferred). If an operator realises too late, they can still pause downstream tasks to hold the workflow.

Optionally, we could allow pausing a running task with an explicit choice: either kill the running process (same as manual fail) or let it finish but discard its result and keep the state as paused. I'd leave that for a follow-up.

2. Resume behaviour: Mirror the "clear task" flow exactly — on resume, state goes paused → scheduled and the scheduler picks it up. try_number increments, task runs from the start. No attempt at mid-execution resumption; that's the operator's responsibility (idempotency, deferrable operators, etc.).

The goal is operational transparency: a distinct state and audit trail (paused_by, pause_reason) instead of overloading failed + clear for planned holds.