Skip to content

Commit 92e2072

Browse files
author
Brandon
committed
[emr] document why the bootstrap targets Python 3.11
EMR 7.x officially ships only Python 3.9 and 3.11; its default python3 is 3.9, too old for Hail. Explain in the EMR_PYSPARK_PYTHON constant and the user docs why the bootstrap installs into 3.11 and how to update the target if a future EMR release ships a newer supported Python.
1 parent 3ada1dc commit 92e2072

2 files changed

Lines changed: 13 additions & 5 deletions

File tree

hail/python/hail/docs/cloud/amazon_web_services.rst

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,8 +50,13 @@ Choosing an EMR release
5050
Hail requires Spark 3.5.x. The default release, ``emr-7.3.0``, provides Spark 3.5.3, which matches
5151
the Spark version Hail is built against. If you pass a ``--release-label`` that ships a different
5252
Spark minor version, ``hailctl emr start`` refuses to start the cluster; an unrecognized label
53-
produces a warning. Hail requires Python 3.10 or newer; the bootstrap installs Hail into Python
54-
3.11, which ships with EMR releases 7.1.0 and newer, so use ``emr-7.1.0`` or later.
53+
produces a warning.
54+
55+
Hail also requires a supported version of Python. The EMR 7.x application stack officially ships
56+
only Python 3.9 and 3.11, and its default ``python3`` is 3.9, which is too old for Hail. The
57+
bootstrap therefore installs Hail into Python 3.11 (available on EMR releases 7.1.0 and newer) and
58+
points Spark's driver and executors at it, so use ``emr-7.1.0`` or later. If a future EMR release
59+
ships a newer default Python that Hail supports, ``hailctl emr`` will be updated to target it.
5560

5661
Advanced cluster options
5762
------------------------

hail/python/hailtop/hailctl/emr/start.py

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,12 @@
1717
DEFAULT_EMR_RELEASE = 'emr-7.3.0'
1818
HAIL_REQUIRED_SPARK_MINOR = '3.5'
1919
HAIL_JAR_PATH = '/usr/lib/hail/hail-all-spark.jar'
20-
# EMR 7.x's system python3 is 3.9, but Hail requires Python >= 3.10. The
21-
# bootstrap installs Hail into Python 3.11 (shipped with EMR 7.1+); PySpark
22-
# (driver and executors) must use the same interpreter.
20+
# EMR 7.x's system python3 is 3.9, but Hail requires Python >= 3.10. The EMR 7.x
21+
# application stack officially ships only Python 3.9 and 3.11 (see the release's
22+
# component versions); 3.12 is not provided by default. So the bootstrap installs
23+
# Hail into Python 3.11, and PySpark (driver and executors) must use that same
24+
# interpreter. If a future EMR release ships a newer default Python that Hail
25+
# supports, update this single constant (and the bootstrap's dnf line).
2326
EMR_PYSPARK_PYTHON = '/usr/bin/python3.11'
2427

2528

0 commit comments

Comments
 (0)