exporting keyframe poses and reconstructing RGBD point clouds in Open3D

[elo-zed]

Question about exporting keyframe poses and reconstructing RGBD point clouds in Open3D

Hi,

I'm using RTAB-Map with RGB-D data captured from an Intel RealSense D405. The reconstruction shown in RTAB-Map is very good, but I'm having trouble reproducing the same result in Open3D using the exported keyframes and poses.

Dataset

Camera: Intel RealSense D405

Calibration exported by RTAB-Map:

camera_matrix:
   data: [437.56, 0., 424.37,
          0., 436.316, 250.56,
          0., 0., 1.]

local_transform:
   data: [0., 0., 1., 0.,
          -1., 0., 0., 0.,
          0., -1., 0., 0.]

Depth images are uint16 PNGs, and depth values are around 1000 at approximately 1 meter distance.

Example pose exported by RTAB-Map:

[[ 0.          0.          1.          0.        ]
 [-1.          0.          0.          0.        ]
 [ 0.         -1.          0.          0.        ]
 [ 0.          0.          0.          1.        ]]

What I am doing

I export:

• Keyframe RGB images
• Keyframe depth images
• Keyframe poses

Then in Open3D:

rgbd = o3d.geometry.RGBDImage.create_from_color_and_depth(
    color,
    depth,
    depth_scale=1000.0,
    depth_trunc=0.6,
    convert_rgb_to_intensity=False
)

pcd = o3d.geometry.PointCloud.create_from_rgbd_image(
    rgbd,
    intrinsic
)

pcd.transform(pose)

I merge all keyframe point clouds using the exported poses.

Problem

The reconstruction obtained in Open3D is significantly different from what RTAB-Map shows in the 3D view.

RTAB-Map reconstruction looks correct, while the Open3D reconstruction is distorted/misaligned even though the same keyframes and poses are used.

Questions

1. What exactly is the coordinate convention of the exported poses?

   • Camera → World ?
   • World → Camera ?

2. Should the exported local_transform be applied when reconstructing point clouds outside RTAB-Map?

3. When exporting keyframe images and poses, is there a guaranteed one-to-one correspondence between:

   • exported RGB image
   • exported depth image
   • exported pose

4. For RealSense D405 depth images stored as uint16 (values around 1000 for 1 meter), what is the correct depth scaling expected by RTAB-Map internally?

5. Is there an example showing how to reconstruct the same point cloud outside RTAB-Map using:

   • exported RGB images
   • exported depth images
   • exported poses

so that the result matches the RTAB-Map viewer?

Thanks!

[matlabbe]

What exactly is the coordinate convention of the exported poses?
Should the exported local_transform be applied when reconstructing point clouds outside RTAB-Map?

It depends, you can export the poses in different frames. By default, the pose would be map->base. In that case, you would need to apply local_transform to it to get map->base->camera. You can also export in camera frame directly to avoid to manually multiply the transform yourself.

A side note, RTAB-Map follows these conventions: https://www.ros.org/reps/rep-0103.html#coordinate-frame-conventions

local_transform:
   data: [0., 0., 1., 0.,
          -1., 0., 0., 0.,
          0., -1., 0., 0.]

this is base->camera transform, which is only the optical rotation in this case. For the actual poses, I don't see how you exported them in your post. In rtabmap, you can do "File->Export poses...".

When exporting keyframe images and poses, is there a guaranteed one-to-one correspondence between:

Should be by default. Otherwise, there is a way to match them afterwards, like exporting with ID or stamp.

For RealSense D405 depth images stored as uint16 (values around 1000 for 1 meter), what is the correct depth scaling expected by RTAB-Map internally?

unin16 format is expected to be in mm (1000 mm = 1m), which 32FC1 format is expected to be in meters.

Is there an example showing how to reconstruct the same point cloud outside RTAB-Map using:

I don't have an example specially, though we have this example about bundler format and MeshLab: https://github.com/introlab/rtabmap/wiki/Export-Raster-Layers-to-MeshLab-(Bundler-format)