feat(docker): build Python API wheels for 3.8-3.14 in the dev container#9714
Open
JArmandoAnaya wants to merge 4 commits into
Open
feat(docker): build Python API wheels for 3.8-3.14 in the dev container#9714JArmandoAnaya wants to merge 4 commits into
JArmandoAnaya wants to merge 4 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Updates CARLA’s Docker-based UE5 development environment to default to Ubuntu 24.04 while keeping Ubuntu 22.04 selectable via --ubuntu-distro, aligning the container workflow with the project’s host-side 24.04 support.
Changes:
- Split Dockerfiles into per-distro directories (
Util/Docker/22.04/andUtil/Docker/24.04/) and default scripts to 24.04. - Reworked
build.shto use a named build context (carla-root) instead of staging requirements into.tmp/. - Updated docs and changelog to reflect the new default and distro selection.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| Util/Docker/run.sh | Default --ubuntu-distro to 24.04 and update help text. |
| Util/Docker/build.sh | Default to 24.04, select per-distro Dockerfiles, and use named build context for requirements. |
| Util/Docker/24.04/Base.Dockerfile | Add new 24.04 base image with apt CMake, pip behavior for PEP 668, and CPython 3.10 build. |
| Util/Docker/24.04/Development.Dockerfile | Add new 24.04 dev image with UID/GID mapping and docker GID collision guard. |
| Util/Docker/24.04/build_scripts/build_cpython.sh | New helper to build/install CPython 3.10 via altinstall. |
| Util/Docker/22.04/Base.Dockerfile | Move 22.04 base to per-distro directory; switch to libtiff-dev; stop bundling newer CMake. |
| Util/Docker/22.04/Development.Dockerfile | Move 22.04 dev Dockerfile; switch requirements to named context copies. |
| Docs/build_linux_ue5.md | Update supported Ubuntu versions and recommend Docker for version validation. |
| Docs/build_devcontainer.md | Update devcontainer docs for 24.04 default and 22.04 opt-in. |
| CHANGELOG.md | Add user-facing entry describing new default and Python 3.10-in-24.04 image behavior. |
Comments suppressed due to low confidence (1)
Docs/build_devcontainer.md:8
- The build scripts now rely on BuildKit named build contexts (
--build-context/COPY --from=carla-root). It would help to mention this in prerequisites (e.g., “Docker with BuildKit enabled / sufficiently recent Docker Engine”) so contributors on older Docker installs can self-diagnose build failures quickly.
CARLA UE5 can be built inside a Docker container running Ubuntu 24.04 (default) or Ubuntu 22.04. This is useful when your host OS does not meet UE5's compilation requirements, or when you want to validate the build on a specific Ubuntu version without changing your host. The container provides the correct toolchain and dependencies while you keep all source code on the host.
## Prerequisites
- [Docker](https://docs.docker.com/engine/install/) installed and configured for non-root use
- [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) for GPU support
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…table The Util/Docker dev environment isolates the CARLA UE5 toolchain (CMake, ninja, clang, Python, libtiff, libpng, glibc shims, UE5 prerequisites) from whatever the host happens to have installed. That isolation gives contributors three things that matter in practice: 1. Host independence: building CARLA does not require uninstalling or shadowing the system Python, system CMake, or other versions a host distro and other projects already rely on. 2. Reproducibility: a build or test failure inside the image reproduces bit-for-bit on a different machine by rebuilding the same image, instead of turning into a per-host investigation of CMake versions and LD_LIBRARY_PATH state. 3. Stable compilation across many computers: the recipe travels unchanged across Ubuntu 22.04, Ubuntu 24.04, Fedora, Pop_OS, etc., which keeps "works on my machine" from becoming a per-developer scavenger hunt. PR carla-simulator#9596 added host-side Ubuntu 24.04 build support (the __isoc23_strtol glibc shim, the libtiff5-dev to libtiff-dev swap, and the PEP 668 pip handling) but deferred the Docker side. This PR closes the developer-facing half of that gap so a contributor on a 24.04 host no longer gets a 22.04 image whose Base.Dockerfile fails on a missing libtiff5-dev package. Layout * The two Dockerfiles that the helper scripts actually invoke are reorganised into a per-distro layout: Util/Docker/22.04 and Util/Docker/24.04, each holding a linear, conditional-free Base.Dockerfile and Development.Dockerfile. Util/Docker/build.sh resolves --ubuntu-distro to the matching subdirectory and refuses values with no directory; Util/Docker/run.sh defaults to the ue5-24.04 image and container names. * Util/Docker/Release.Dockerfile has no caller anywhere in the repo (build.sh, run.sh, .github/workflows, Util/Tools/Deploy.{sh,bat}, the CMake graph) and is left untouched. Cleaning up that orphan is out of scope for this PR. * build.sh passes --build-context carla-root=<repo root> so each Dockerfile reads requirements.txt and PythonAPI/{examples,util}/ requirements.txt directly via COPY --from=carla-root, replacing the previous .tmp staging copy. 24.04 image * Base.Dockerfile installs libtiff-dev (the noble package name), upgrades pip with --ignore-installed (the dpkg-managed pip on noble has no RECORD file so a normal self-upgrade fails), and compiles Python 3.10.14 from source via Util/Docker/24.04/build_scripts/ build_cpython.sh so the Python API can target either 3.10 or 3.12 via -DPython_ROOT_DIR. CMake comes from apt and resolves to 3.28.3, which clears CARLA's current 3.27.2 minimum. * Development.Dockerfile reuses an existing group when the host docker GID collides with systemd-journal at GID 999 inside the noble base image, instead of failing groupadd. 22.04 image * Base.Dockerfile keeps its previous shape minus the upstream CMake tarball download. apt cmake on jammy is 3.22.1, below the project minimum, so users opting back into 22.04 must upgrade CMake themselves; the Dockerfile keeps a single apt install cmake line and the 22.04 path becomes best-effort. Docs and CHANGELOG * Docs/build_devcontainer.md documents the new default, adds a "Build for a different Ubuntu version" subsection, and updates every docker exec example plus the devcontainer.json snippet. * Docs/build_linux_ue5.md no longer says "Ubuntu 22.04 at minimum", it lists 22.04 and 24.04 as supported and points at the Docker fallback for any other host. * CHANGELOG.md gains one user-facing bullet. Out of scope CI workflow changes are intentionally not included. PR carla-simulator#9601 owns the _ci-ubuntu.yml matrix and is blocked on the carlasim/carla-builder image being available, which this PR produces locally so the same recipe can be published. Validated on Ubuntu 24.04 (RTX 50, driver 590.48): carla-base and carla-development images build clean, libcarla_test_server 44/44 and libcarla_test_client 58/58 pass, cmake --build Build --target package succeeds with both Python 3.10 and Python 3.12 wheels. Related: carla-simulator#9596 (host-side 24.04, merged), carla-simulator#9597 (umbrella issue, "24.04 CI image" checkbox), carla-simulator#9601 (CI matrix, open).
JArmandoAnaya
changed the title
Add a carla-python-api-wheels CMake target and a CARLA_PYTHON_API_VERSIONS option that build the CARLA Python API wheel for one or many interpreters in a single step. By default it builds only the currently configured interpreter (unchanged behaviour); given a version list it locates each python3.X, builds the ones present, and repairs each to a portable manylinux tag with auditwheel (best effort). Missing or out-of-range versions are reported but never stop the run, which fails only when no wheel could be built at all. Because the Python API links a version-specific Boost.Python target resolved at configure time, every non-current interpreter is built in its own nested configure. Both find_package(Python) and find_package(Python3) are pinned to the same interpreter so Boost and CARLA never disagree on the version. Provision the dev container to back this end to end: the Ubuntu 24.04 and 22.04 images now ship Python 3.8 through 3.14. The distro's own interpreter (3.12 on noble, 3.10 on jammy) stays the default and is provisioned in place; the other versions are compiled from source. A contributor can therefore produce wheels for any supported interpreter from any Linux host through the container, regardless of the host distribution. - CMake/BuildPythonApiWheels.cmake: per-version build and auditwheel repair - CMake/Options.cmake, PythonAPI/CMakeLists.txt: the target and its options - Util/Docker: per-distro interpreter provisioning and build requirements - Docs/build_linux_ue5.md, Docs/build_devcontainer.md: usage and prerequisites
JArmandoAnaya
force-pushed
the
feat/docker-default-ubuntu-24-04
branch
from
5ccabf2 to
5e14187
Compare
Use environment markers so a single pip install -r requirements.txt succeeds on every interpreter from 3.8 to 3.14 and installs the maximal supported subset per version, instead of hard-failing on out-of-range packages. - pygame for < 3.14, pygame-ce (a drop-in that exposes import pygame) for >= 3.14, which has no mainline pygame wheel yet; this keeps manual_control and the other pygame examples runnable on 3.14. - Raise the open3d marker to <= 3.12 (wheels exist through cp312) and declare matplotlib, which open3d_lidar imports directly. - Gate invertedai with its own >= 3.10, < 3.13 range so pip skips it out of range rather than erroring.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The dev container can now build the CARLA Python API wheel for every interpreter from Python 3.8 to 3.14, including 3.12, in a single command. The important part is where it runs: inside the container, so the host distribution no longer matters. You can be on Fedora, Arch, a minimal CI runner, or an Ubuntu release CARLA does not officially support (for example a future 26.04), and still produce the full set of wheels. This is the main value of the change. It turns "build CARLA's Python bindings" from a host setup problem into a one line container build that works the same everywhere.
This builds on top of moving the dev environment default to Ubuntu 24.04 (the original scope of this PR, see #9596 for the host side). Ubuntu 22.04 stays available with
--ubuntu-distro 22.04.Under the hood, a new
carla-python-api-wheelsCMake target and aCARLA_PYTHON_API_VERSIONSoption drive the build. By default nothing changes: it builds only the interpreter you configured, exactly likecarla-python-apitoday. Give it a version list and it builds a wheel for each interpreter it can find and repairs each one to a portable manylinux tag. Because the Python API links a version specific Boost.Python library, each extra interpreter needs its own configure step, which the target handles for you.Changes
carla-python-api-wheelstarget andCARLA_PYTHON_API_VERSIONSoption (CMake/BuildPythonApiWheels.cmake,CMake/Options.cmake,PythonAPI/CMakeLists.txt). Default builds the current interpreter only. A version list builds eachpython3.Xthat is present, reports and skips the ones that are missing or out of range, and only fails if no wheel could be built at all. Each wheel is repaired to a manylinux tag with auditwheel (best effort, falls back to the plain wheel).Build/PythonAPI/dist/, the same directorycarla-python-apialready uses for the single interpreter wheel, with exactly one wheel per version. The current interpreter builds straight into that directory, so after a successful repair the unrepaired wheel is removed and only the manylinux build is kept.find_package(Python)andfind_package(Python3)to the same interpreter, so Boost.Python and the CARLA bindings never resolve different versions.Util/Docker/22.04/andUtil/Docker/24.04/now ship Python 3.8 through 3.14. The distro's own interpreter stays the default (3.12 on noble, 3.10 on jammy) and is provisioned in place; the remaining versions are compiled from source.PythonAPI/examples/requirements.txtnow uses environment markers so a singlepip install -r requirements.txtworks on every interpreter from 3.8 to 3.14 and installs the maximal supported subset instead of hard failing.pygamefor< 3.14andpygame-ce(a drop-in that exposesimport pygame) for>= 3.14, which has no mainlinepygamewheel yet, somanual_controland the other pygame examples run on 3.14.open3draised to<= 3.12(wheels exist through cp312) withmatplotlibdeclared foropen3d_lidar.invertedaigated to its own>= 3.10, < 3.13range so pip skips it out of range rather than erroring.build.shandrun.shis24.04, with--ubuntu-distro 22.04to opt back. The 24.04 image also installs a CMake new enough for the project minimum.Docs/build_devcontainer.md,Docs/build_linux_ue5.md) with a worked multi version example and the per interpreter prerequisites. One user facingCHANGELOG.mdbullet.Where has this been tested?
manylinuxtag on both the Ubuntu 24.04 and 22.04 images. Each wheel was then installed into a clean per version environment,import carlaconfirmed, the examplesrequirements.txtinstalled, andPythonAPI/examples/manual_control.pyrun against the live server. The matrix below was proven on real interpreters:import carlamanual_control.pyPossible Drawbacks
pygame-cebecause mainlinepygamehas no 3.14 wheel yet. It is a drop-in, soimport pygameis unchanged.open3d(used only byopen3d_lidar) andinvertedai(used only byinvertedai_traffic) keep their upstream support windows, so those two examples are unavailable on the newest interpreters.Related: #9596, #9597, #9601.
This change is