Debian-docker: Use rosinstall_generator for dependency resolution (#685)

* Use rosinstall_generator for dependency resolution

* Add badges for other platforms

* Resort README

* TMP: don't use cache

* Fix paths

* Revert "TMP: don't use cache"

This reverts commit d48eb22.

* Fix bash->sh script

* Fix exclude path

* Update comments

* Exclude mujoco and gz

* Fix ""

* Exclude ign

* Explicitely exclude ros_gz*

* Don't build rviz2

* Only add deps to repos file

* Only build pinocchio in the end

* Don't build eigenpy (not needed by pinocchio with BUILD_PYTHON_INTERFACE=OFF

* Skip gz_ros2_control_demos  (implies gz_sim_vendor)

* Don't build hpp-fcl

* Remove release flag

* Skip eigenpy and hpp-fcl already from rosinstall-generator

* Skip the gz demos in the stack build

Author: christophfroehlich

.github/workflows/build_and_publish_debian_docker.yaml

@@ -13,6 +13,7 @@ on:
     paths:
       - '.github/workflows/build_and_publish_debian_docker.yaml'
       - 'ros2_debian/**'
+      - 'ros_controls.*.repos'
   schedule:
     - cron: '0 1 * * MON'
 
@@ -33,16 +34,16 @@ jobs:
         include:
           - ros_distro: 'humble'
             debian_version: 'debian11'
-            source_packages: filters xacro diagnostic_updater backward_ros generate_parameter_library angles ackermann_msgs pal_statistics topic_tools
+            exclude_packages: ''
           - ros_distro: 'jazzy'
             debian_version: 'debian12'
-            source_packages: filters xacro diagnostic_updater backward_ros generate_parameter_library angles ackermann_msgs sdformat_urdf pal_statistics pinocchio topic_tools
+            exclude_packages: ''
           - ros_distro: 'kilted'
             debian_version: 'debian12'
-            source_packages: filters xacro diagnostic_updater backward_ros generate_parameter_library angles ackermann_msgs sdformat_urdf pal_statistics pinocchio topic_tools
+            exclude_packages: ''
           - ros_distro: 'rolling'
             debian_version: 'debian12'
-            source_packages: filters xacro diagnostic_updater backward_ros generate_parameter_library angles ackermann_msgs sdformat_urdf pal_statistics pinocchio topic_tools
+            exclude_packages: ''
 
     steps:
       - uses: actions/checkout@v6
@@ -63,7 +64,7 @@ jobs:
       - name: Build and push image
         uses: docker/build-push-action@v6
         with:
-          context: ros2_debian
+          context: .
           push: true
           provenance: false
           sbom: false
@@ -77,7 +78,7 @@ jobs:
           cache-to: type=gha,scope=debian-${{ matrix.ros_distro }},mode=max
           build-args: |
             ROS_DISTRO=${{ matrix.ros_distro }}
-            SOURCE_PACKAGES=${{ matrix.source_packages }}
+            EXCLUDE_PACKAGES=gz_ros2_control gz_ros2_control_demos ign_ros2_control mujoco_ros2_control ros_gz_sim ros_gz_bridge rviz2 eigenpy hpp-fcl ${{ matrix.exclude_packages }}
 
   stack-build:
     if: ${{ github.event_name == 'pull_request' }}

README.md

@@ -9,13 +9,12 @@ Within this repo the full ros2_control stack is built and tested once per day.
 
 ## Released versions
 
-[![Rolling Stack Build](https://github.com/ros-controls/ros2_control_ci/actions/workflows/rolling-binary-build.yml/badge.svg)](https://github.com/ros-controls/ros2_control_ci/actions/workflows/rolling-binary-build.yml)
-
-[![Kilted Stack Build](https://github.com/ros-controls/ros2_control_ci/actions/workflows/kilted-binary-build.yml/badge.svg)](https://github.com/ros-controls/ros2_control_ci/actions/workflows/kilted-binary-build.yml)
-
-[![Jazzy Stack Build](https://github.com/ros-controls/ros2_control_ci/actions/workflows/jazzy-binary-build.yml/badge.svg)](https://github.com/ros-controls/ros2_control_ci/actions/workflows/jazzy-binary-build.yml)
-
-[![Humble Stack Build](https://github.com/ros-controls/ros2_control_ci/actions/workflows/humble-binary-build.yml/badge.svg)](https://github.com/ros-controls/ros2_control_ci/actions/workflows/humble-binary-build.yml)
+| Platform | Build status |
+| --- | --- |
+| Ubuntu | [![Rolling Stack Build](https://github.com/ros-controls/ros2_control_ci/actions/workflows/rolling-binary-build.yml/badge.svg)](https://github.com/ros-controls/ros2_control_ci/actions/workflows/rolling-binary-build.yml) [![Kilted Stack Build](https://github.com/ros-controls/ros2_control_ci/actions/workflows/kilted-binary-build.yml/badge.svg)](https://github.com/ros-controls/ros2_control_ci/actions/workflows/kilted-binary-build.yml) [![Jazzy Stack Build](https://github.com/ros-controls/ros2_control_ci/actions/workflows/jazzy-binary-build.yml/badge.svg)](https://github.com/ros-controls/ros2_control_ci/actions/workflows/jazzy-binary-build.yml) [![Humble Stack Build](https://github.com/ros-controls/ros2_control_ci/actions/workflows/humble-binary-build.yml/badge.svg)](https://github.com/ros-controls/ros2_control_ci/actions/workflows/humble-binary-build.yml) |
+| Debian | [![Debian Stack Build](https://github.com/ros-controls/ros2_control_ci/actions/workflows/debian-stack-build.yml/badge.svg)](https://github.com/ros-controls/ros2_control_ci/actions/workflows/debian-stack-build.yml) |
+| RHEL | [![RHEL Stack Build](https://github.com/ros-controls/ros2_control_ci/actions/workflows/rhel-stack-build.yml/badge.svg)](https://github.com/ros-controls/ros2_control_ci/actions/workflows/rhel-stack-build.yml) |
+| Windows | [![Windows Stack Build](https://github.com/ros-controls/ros2_control_ci/actions/workflows/windows-stack-build.yml/badge.svg)](https://github.com/ros-controls/ros2_control_ci/actions/workflows/windows-stack-build.yml) |
 
 ## Compatibility versions
 We thrive to make the rolling development version of the ros2_control stack compatible with earlier releases of ROS 2. This is done by building the rolling version of the stack from source with the earlier releases of ROS 2.
@@ -26,15 +25,16 @@ We thrive to make the rolling development version of the ros2_control stack comp
 
 [![Check Rolling Compatibility on Humble with Stack Build](https://github.com/ros-controls/ros2_control_ci/actions/workflows/rolling-compatibility-humble-binary-build.yml/badge.svg)](https://github.com/ros-controls/ros2_control_ci/actions/workflows/rolling-compatibility-humble-binary-build.yml)
 
-## ros2_ubuntu
+## Docker images
+### ros2_ubuntu
 Extension of the [official OSRF docker images](https://github.com/osrf/docker_images), but with versions for `ros2`(main) and `ros2-testing` binaries.
 
-## ros2_rhel
+### ros2_rhel
 This is a sample Dockerfile that show ros2 installed on almalinux based on the [binary installation instructions at docs.ros.org](https://docs.ros.org/en/rolling/Installation/RHEL-Install-RPMs.html). The most recent public build can be found with the docker tag: `ghcr.io/ros-controls/ros:<ROS_DISTRO>-rhel` where `<ROS_DISTRO>` is replaced with the ros distribution you are targeting.
 
 The image tries to emulate the official ros2 images but on alma (downstream RHEL) linux. Therefore a few extra packages can be found within this repo that are not mentioned at the above link, i.e. colcon, etc.
 
-## ros2_debian
+### ros2_debian
 Provides docker images for building ros2_control on debian.
 
 Pull and run the images with:

ros2_debian/Dockerfile.debian11

@@ -4,6 +4,7 @@ FROM ${from}
 
 ARG CMAKE_INSTALL_PREFIX=/usr/local
 ARG NUM_THREADS=8
+ARG EXCLUDE_PACKAGES=""
 
 # Avoid warnings by switching to noninteractive
 ENV DEBIAN_FRONTEND=noninteractive
@@ -142,43 +143,58 @@ RUN \
 # ---- ROS ----
 # clone source, ROS core
 ENV ROS2_WS=/opt/ros2_ws
-ARG SOURCE_PACKAGES=""
 WORKDIR $ROS2_WS
+COPY ros_controls.*.repos /tmp/
 RUN \
-  : "build ROS core from source" && \
-  mkdir -p src/ros2 && \
-  vcs import --input https://raw.githubusercontent.com/ros2/ros2/$ROS_DISTRO/ros2.repos src/ros2 && \
-  if [ -n "$SOURCE_PACKAGES" ]; then \
-    # get the release repos from gpb and its dependencies not included in ros2.repos
-    for i in {1..5}; do \
-      rosinstall_generator $SOURCE_PACKAGES --exclude-path src/ros2 --rosdistro $ROS_DISTRO --deps \
-        > deps.repos && break || echo "retry #${i} .." && sleep 60; \
-    done && \
-    mkdir -p src/additional_pkgs && \
-    vcs import src/additional_pkgs < deps.repos && \
-    apt-get update -y -qq && \
-    # install only dependencies for additional_pkgs, not for ros2.repos
-    ROS_PACKAGE_PATH=$ROS2_WS rosdep install -iyr --from-path src/additional_pkgs || true ; \
+  : "build dependencies for ros-controls stack from release repos" && \
+  repo_file=/tmp/ros_controls.${ROS_DISTRO}.repos && \
+  if [ ! -f "$repo_file" ]; then \
+    echo "Missing repos file: $repo_file" >&2; \
+    exit 1; \
   fi && \
-  # build source, ROS core
+  mkdir -p src/imported_pkgs && \
+  vcs import src/imported_pkgs < "$repo_file" && \
+  TARGET_PACKAGES=$(colcon list --base-paths src/imported_pkgs --names-only | tr '\n' ' ') && \
+  if [ -z "$TARGET_PACKAGES" ]; then \
+    echo "No packages found in imported repos" >&2; \
+    exit 1; \
+  fi && \
+  EXCLUDE_ARGS="" && \
+  if [ -n "$EXCLUDE_PACKAGES" ]; then \
+    EXCLUDE_ARGS="--exclude $EXCLUDE_PACKAGES"; \
+  fi && \
+  success=0 && \
+  for i in 1 2 3 4 5; do \
+    if rosinstall_generator $TARGET_PACKAGES --rosdistro $ROS_DISTRO --deps --deps-only --format repos $EXCLUDE_ARGS > deps.repos; then \
+      success=1; \
+      break; \
+    fi; \
+    echo "retry #${i} .."; \
+    sleep 60; \
+  done && \
+  if [ "$success" -ne 1 ] || [ ! -s deps.repos ]; then \
+    echo "Failed to generate deps.repos" >&2; \
+    exit 1; \
+  fi && \
+  rm -rf src/imported_pkgs && \
+  mkdir -p src/generated && \
+  vcs import src/generated < deps.repos && \
+  apt-get update -y -qq && \
+  ROS_PACKAGE_PATH=$ROS2_WS rosdep install -iyr --from-path src/generated || true && \
+  # build generated stack
   colcon build \
   --mixin release build-testing-off \
-  --cmake-args --no-warn-unused-cli \
-  --packages-up-to robot_state_publisher tf2_ros tf2_eigen tf2_kdl tf2_eigen_kdl yaml_cpp_vendor filters \
-  ros2param ros2interface ros2topic ros2action ros2lifecycle ros2launch ros2run ros_testing \
-  xacro diagnostic_updater generate_parameter_library angles example_interfaces \
-  backward_ros pal_statistics topic_tools \
-  ackermann_msgs trajectory_msgs tf2_msgs tf2_geometry_msgs sensor_msgs geometry_msgs nav_msgs && \
-  rm -rf log src build *.repos
+  --cmake-args --no-warn-unused-cli && \
+  rm -rf log src build *.repos /tmp/ros_controls.*.repos
 
 # add default.yaml to the image
-COPY defaults.yaml /root/.colcon/defaults.yaml
+COPY ros2_debian/defaults.yaml /root/.colcon/defaults.yaml
 
 # there is no python3-graphviz on debian11, so install it via pip
 RUN pip3 install --no-cache-dir graphviz
 
 # Setup entrypoint (sourcing of ROS)
-COPY ./ros_entrypoint.sh /
+COPY ros2_debian/ros_entrypoint.sh /
 RUN chmod +x /ros_entrypoint.sh
 ENTRYPOINT [ "/ros_entrypoint.sh" ]
 CMD ["bash"]

ros2_debian/Dockerfile.debian12

@@ -3,6 +3,7 @@ FROM ${from}
 
 ARG CMAKE_INSTALL_PREFIX=/usr/local
 ARG NUM_THREADS=8
+ARG EXCLUDE_PACKAGES=""
 
 # Avoid warnings by switching to noninteractive
 ENV DEBIAN_FRONTEND=noninteractive
@@ -151,62 +152,71 @@ RUN \
 # clone source, ROS core
 ENV ROS2_WS=/opt/ros2_ws
 WORKDIR $ROS2_WS
-ARG SOURCE_PACKAGES=""
+COPY ros_controls.*.repos /tmp/
 RUN \
-  : "build ROS core from source" && \
+  : "build dependencies for ros-controls stack from release repos" && \
   . "$HOME/.cargo/env" && \
-  mkdir -p src/ros2 && \
-  vcs import --input https://raw.githubusercontent.com/ros2/ros2/$ROS_DISTRO/ros2.repos src/ros2 && \
-  if [ -n "$SOURCE_PACKAGES" ]; then \
-    # get the release repos from gpb and its dependencies not included in ros2.repos
-    for i in {1..5}; do \
-      rosinstall_generator $SOURCE_PACKAGES --exclude-path src/ros2 --rosdistro $ROS_DISTRO --deps \
-        > deps.repos && break || echo "retry #${i} .." && sleep 60; \
-    done && \
-    mkdir -p src/additional_pkgs && \
-    vcs import src/additional_pkgs < deps.repos && \
-    apt-get update -y -qq && \
-    # install only dependencies for additional_pkgs, not for ros2.repos
-    ROS_PACKAGE_PATH=$ROS2_WS rosdep install -iyr --from-path src/additional_pkgs || true ; \
+  repo_file=/tmp/ros_controls.${ROS_DISTRO}.repos && \
+  if [ ! -f "$repo_file" ]; then \
+    echo "Missing repos file: $repo_file" >&2; \
+    exit 1; \
   fi && \
-  if [ $ROS_DISTRO = "rolling" ]; then \
-    apt-get update -y -qq && \
-    # we don't want to install all system dependencies for ros2.repos
-    # https://github.com/ros2/geometry2/pull/826
-    # https://github.com/ros2/launch/pull/817
-    apt-get -y install --no-install-recommends liborocos-kdl-dev liborocos-kdl1.5 python3-osrf-pycommon ;\
+  mkdir -p src/imported_pkgs && \
+  vcs import src/imported_pkgs < "$repo_file" && \
+  TARGET_PACKAGES=$(colcon list --base-paths src/imported_pkgs --names-only | tr '\n' ' ') && \
+  if [ -z "$TARGET_PACKAGES" ]; then \
+    echo "No packages found in imported repos" >&2; \
+    exit 1; \
   fi && \
-  # build source, ROS core
-  colcon build \
-  --mixin release build-testing-off \
-  --cmake-args --no-warn-unused-cli -DCMAKE_CXX_FLAGS="-Wno-maybe-uninitialized"  \
-  --packages-up-to robot_state_publisher tf2_ros tf2_eigen tf2_kdl tf2_eigen_kdl yaml_cpp_vendor filters \
-  ros2param ros2interface ros2topic ros2action ros2lifecycle ros2launch ros2run ros_testing \
-  xacro diagnostic_updater generate_parameter_library angles example_interfaces \
-  backward_ros pal_statistics topic_tools ros_environment \
-  ackermann_msgs trajectory_msgs tf2_msgs tf2_geometry_msgs sensor_msgs geometry_msgs nav_msgs \
-  sdformat_urdf && \
-  # build pinocchio from source with different cmake args
-  . install/setup.sh && \
+  EXCLUDE_ARGS="" && \
+  if [ -n "$EXCLUDE_PACKAGES" ]; then \
+    EXCLUDE_ARGS="--exclude $EXCLUDE_PACKAGES"; \
+  fi && \
+  success=0 && \
+  for i in 1 2 3 4 5; do \
+    if rosinstall_generator $TARGET_PACKAGES --rosdistro $ROS_DISTRO --deps --deps-only --format repos $EXCLUDE_ARGS > deps.repos; then \
+      success=1; \
+      break; \
+    fi; \
+    echo "retry #${i} .."; \
+    sleep 60; \
+  done && \
+  if [ "$success" -ne 1 ] || [ ! -s deps.repos ]; then \
+    echo "Failed to generate deps.repos" >&2; \
+    exit 1; \
+  fi && \
+  rm -rf src/imported_pkgs && \
+  mkdir -p src/generated && \
+  vcs import src/generated < deps.repos && \
+  apt-get update -y -qq && \
+  ROS_PACKAGE_PATH=$ROS2_WS rosdep install -iyr --from-path src/generated || true && \
+  # build generated stack, then build pinocchio with special flags when package is present
   colcon build \
-    --mixin release build-testing-off \
-    --cmake-args --no-warn-unused-cli -DBUILD_EXAMPLES=OFF -DBUILD_PYTHON_INTERFACE=OFF \
-      -DBUILD_WITH_COLLISION_SUPPORT=OFF -DBUILD_WITH_HPP_FCL_SUPPORT=OFF \
-    --base-path src/additional_pkgs --packages-up-to pinocchio && \
+    --mixin build-testing-off \
+    --cmake-args --no-warn-unused-cli -DCMAKE_CXX_FLAGS="-Wno-maybe-uninitialized" \
+    --packages-skip pinocchio && \
+  if colcon list --base-paths src/generated --names-only | grep -qx pinocchio; then \
+    . install/setup.sh && \
+    colcon build \
+      --cmake-args --no-warn-unused-cli -DBUILD_EXAMPLES=OFF -DBUILD_PYTHON_INTERFACE=OFF \
+        -DBUILD_WITH_COLLISION_SUPPORT=OFF -DBUILD_WITH_HPP_FCL_SUPPORT=OFF \
+      --continue-on-error \
+      --packages-select pinocchio; \
+  fi && \
   # cleanup
-  rm -rf log src build *.repos \
+  rm -rf log src build *.repos /tmp/ros_controls.*.repos \
   && \
   : "remove cache" && \
   rm -rf /var/lib/apt/lists/*
 
 # add default.yaml to the image
-COPY defaults.yaml /root/.colcon/defaults.yaml
+COPY ros2_debian/defaults.yaml /root/.colcon/defaults.yaml
 
 # there is no python3-graphviz on debian12, so install it via pip
 RUN pip3 install --no-cache-dir graphviz --break-system-packages
 
 # Setup entrypoint (sourcing of ROS)
-COPY ./ros_entrypoint.sh /
+COPY ros2_debian/ros_entrypoint.sh /
 RUN chmod +x /ros_entrypoint.sh
 ENTRYPOINT [ "/ros_entrypoint.sh" ]
 CMD ["bash"]