hailctl emr: AWS EMR support (CHANGELOG)

[hmkim]

Change Description

Adds hailctl emr, a command group (start / stop / list / submit) for provisioning and running Hail on Amazon EMR, bringing AWS toward parity with hailctl dataproc and hailctl hdinsight.

Design was discussed with the Hail team on the forum first: https://hail.zulipchat.com/#narrow/channel/127634-Feature-Requests/topic/Contributing.20.60hailctl.20emr.60.20.28AWS.20EMR.20support.29/near/612285285

What it does

• Pure-Python Typer module (hailtop/hailctl/emr/) that drives EMR through the boto3 control plane (native Configurations/BootstrapActions, ClusterRunning/StepComplete waiters — no aws CLI shell-out). All s3:// file I/O routes through RouterAsyncFS; boto3 is used for the EMR control plane only.
• One-line change in RouterFS.scala: HAIL_CLOUD=aws returns an all-None CloudStorageConfig, so the router falls back to HadoopFS and EMRFS/hadoop-aws resolve s3:///s3a://. (Per the forum discussion, a native S3StorageFS is intentionally left for the Batch backend and is out of scope here.)
• Bootstrap script shipped in the wheel via package_data, uploaded to the user's S3 scratch bucket at start time; installs Hail from PyPI (pyspark excluded, since EMR provides it).
• EMR-release → Spark compatibility table (patch differences within 3.5.x allowed, minor mismatch is a hard error, unknown label warns). Default release emr-7.3.0 (Spark 3.5.x).
• Installs Hail into Python 3.11 (shipped with EMR 7.1+) and points spark.pyspark.python / PYSPARK_PYTHON there, since EMR's system python3 is 3.9 and Hail requires >= 3.10.
• Two config variables (emr/region, emr/remote_tmpdir), user docs, and a preflight check for the default EMR IAM roles.

Testing

• 62 unit tests (boto3 Stubber/Mock), plus ruff clean.
• Validated on real AWS: the tutorial GWAS on the 1000 Genomes subset (matches the published tutorial, min p 3.5e-9 @ 8:19600329), a full-autosome GWAS (1000G phase 3, ~81M variants × 2,504 samples, through PCA-adjusted regression), and a gnomAD annotation + Parquet export.

Open item (from the forum thread)

• Wheel distribution currently uses pip install hail==$VERSION from PyPI. If the team prefers pushing a wheel to a known bucket (as with Dataproc), the bootstrap is easy to switch. Marked draft pending that decision.

Note: the (hail#XXXXX) reference in the change log entry is a placeholder for this PR's number.

Security Assessment

• This change cannot impact the Hail Batch instance as deployed by Broad Institute in GCP

Impact Rating

• This change has no security impact

Impact Description

New hailctl client-side command group for AWS EMR plus a one-line JVM FS routing arm for HAIL_CLOUD=aws. It adds no code paths to the Hail Batch service deployed in GCP; it only affects users who run hailctl emr against their own AWS accounts with their own credentials.