ROS2 Humble capture pipeline for synchronized data recording from:
- Intel D455 — stereo IR (848x480 @ 30 fps) + IMU (gyro/accel @ 200 Hz)
- Intel D405 — color stream + camera info
- Hand controller — raw frames via CAN (SocketCAN, ID 0x112) or USART (ttyUSB, 16-byte framed). Protocol is selected at launch.
All streams are recorded into a single ros2 bag (mcap format) with a shared ROS clock. Controller frame assembly, filtering, and calibration happen offline in the Python pipeline — the recorder captures raw frames only.
Two colcon packages:
| Package | Type | Contents |
|---|---|---|
umi_dex_msgs |
ament_cmake | Custom message definitions (CanFrame, HandJointState, UsartFrame) |
umi_dex_bringup |
ament_python | Nodes, launch files, config |
| Component | Install |
|---|---|
| Ubuntu 22.04 | — |
| ROS2 Humble | sudo apt install ros-humble-ros-base ros-dev-tools |
| RealSense ROS2 | sudo apt install ros-humble-realsense2-camera or build from source |
SocketCAN (for controller_protocol:=can) |
Kernel built-in; configure with sudo ip link set can0 up type can bitrate 1000000 |
USART / ttyUSB (for controller_protocol:=usart) |
sudo apt install python3-serial; add user to dialout once: sudo usermod -aG dialout $USER && newgrp dialout |
cd ~/ros2_ws/src
git clone https://github.com/IntelRealSense/realsense-ros.git -b development
cd ~/ros2_ws
rosdep install --from-paths src --ignore-src -r -y
colcon build --packages-select realsense2_camera realsense2_camera_msgs realsense2_description
source install/setup.bash# 1. Create (or reuse) a colcon workspace.
mkdir -p ~/ros2_ws/src && cd ~/ros2_ws/src
# 2. Symlink both packages into the workspace.
ln -s /path/to/UMI-Dex/ros2/umi_dex_msgs umi_dex_msgs
ln -s /path/to/UMI-Dex/ros2/umi_dex_bringup umi_dex_bringup
# 3. Build.
cd ~/ros2_ws
colcon build --packages-select umi_dex_msgs umi_dex_bringup
# 4. Source.
source install/setup.bashcolcon test --packages-select umi_dex_bringup runs pytest against the
smoke tests in test/test_imports.py.
ros2 bag record creates the output directory automatically. Default bag_dir is outputs (relative to CWD). Create it before capture if desired:
cd /path/to/UMI-Dex && mkdir -p outputs# Bring up the controller link:
# CAN: sudo ip link set can0 up type can bitrate 1000000
# USART: ensure the user is in `dialout` (one-time: sudo usermod -aG dialout $USER),
# then plug the device (default /dev/ttyUSB0).
# Set camera serials in:
# ros2/umi_dex_bringup/config/camera_serials.conf
# Terminal 1 — hardware streams (cameras + controller; D455 variant)
ros2 launch umi_dex_bringup capture_d455.launch.py # defaults to CAN
ros2 launch umi_dex_bringup capture_d455.launch.py controller_protocol:=usart \
usart_port:=/dev/ttyUSB0 usart_baud:=115200
# Terminal 2 — interactive recorder (needs its own tty for stdin)
ros2 run umi_dex_bringup record_d455.sh --protocol can
ros2 run umi_dex_bringup record_d455.sh --protocol usart
# D455 interactive commands (shown context-sensitively in the recorder's prompt):
# s : start new session (IMU warm-up + episode recording)
# e : start/end episode (within a session)
# c : end session (save bag with all episodes)
# l : list recordings in bag_dir
# r : delete last finished recording
# q : quitA <bag_dir>.session.json sidecar is written alongside the bag directory with provenance anchors (ROS time, wall clock, host info) and episode metadata.
# D455 camera only
ros2 launch umi_dex_bringup d455.launch.py
# D405 camera only
ros2 launch umi_dex_bringup d405.launch.py
# CAN raw frame publisher only
ros2 launch umi_dex_bringup controller.launch.py
# USART raw frame publisher only
ros2 launch umi_dex_bringup controller.launch.py controller_protocol:=usart usart_port:=/dev/ttyUSB0
# Override capture defaults
ros2 launch umi_dex_bringup capture_d455.launch.py can_channel:=can1
# Override recorder defaults
ros2 run umi_dex_bringup record_d455.sh --protocol can --warmup 20 --bag-dir /tmp/bagsros2 launch umi_dex_bringup playback.launch.py bag:=/path/to/bag_dir| Topic | Type | Rate | Source |
|---|---|---|---|
/camera/infra1/image_rect_raw |
sensor_msgs/msg/Image |
30 Hz | realsense2_camera |
/camera/infra1/camera_info |
sensor_msgs/msg/CameraInfo |
30 Hz | realsense2_camera |
/camera/infra2/image_rect_raw |
sensor_msgs/msg/Image |
30 Hz | realsense2_camera |
/camera/infra2/camera_info |
sensor_msgs/msg/CameraInfo |
30 Hz | realsense2_camera |
/camera/imu |
sensor_msgs/msg/Imu |
200 Hz | realsense2_camera |
/camera_d405/color/image_raw |
sensor_msgs/msg/Image |
30 Hz | realsense2_camera |
/camera_d405/color/camera_info |
sensor_msgs/msg/CameraInfo |
30 Hz | realsense2_camera |
/hand/can_raw |
umi_dex_msgs/msg/CanFrame |
~300 Hz | can_raw_node (when controller_protocol=can) |
/hand/usart_raw |
umi_dex_msgs/msg/UsartFrame |
~300 Hz | usart_raw_node (when controller_protocol=usart) |
/session/episode |
std_msgs/msg/String |
event | interactive_capture_node (D455 sessions only) |
Only one of /hand/can_raw or /hand/usart_raw appears in any given bag — the recorder subscribes to the topic matching the selected protocol.
std_msgs/Header header
uint32 arb_id
uint8 dlc
uint8[8] data
std_msgs/Header header
uint16[6] raw # 12-bit encoder counts
uint8 valid_mask # bit i => channel i valid
Raw 16-byte USART packet (55 AA | mask | 6×(lo,hi12) | checksum) emitted by the controller firmware. Already assembled; decoding to joint angles happens offline (or, eventually, inside firmware once calibration moves there).
std_msgs/Header header
string[6] names
float64[6] positions
bool[6] valid
ROS1 support was dropped in v0.3.0 (2026-06). ROS2 Humble is the canonical recorder. The Python offline pipeline still reads legacy ROS1 bags (umi-inspect, umi-extract, umi-slam, umi-process all auto-detect .bag files vs ROS2 bag directories) — so older recordings remain processable.
Apache License 2.0 — see LICENSE.