add data generation code for Habitat-Sim (HM3D, Gibson, MP3D)

Summary:
Add training data generation code for Habitat-Sim (HM3D, Gibson, MP3D)
- `tuple_gen_scannet.sh` generates tuples of RGBD views of scannet (two difficulties).`tuple_gen_hs.sh` generates tuples of RGBD views of all data in habitat-sim (HM3D, Gibson, MP3D). `metadata_gather_scannet.sh` and `metadata_gather_scannet.sh` collects the tuples to form a h5 index file

Reviewed By: dilinwang820

Differential Revision: D74629224

fbshipit-source-id: 7f3538e01051afe8655d389cac4091c200a7edbe

Author: stephenyan1231

README.md

@@ -18,6 +18,7 @@ Please see more visual results and video on our [website](https://mv-dust3rp.git
 
 ## Update Logs
 
+- 2025-5-10: data generation code for Habitat-Sim (HM3D, Gibson, MP3D).
 - 2025-1-30: data generation code of ScanNet.
 - 2025-1-1: A gradio demo, all checkpoints, training/evaluation code and training/evaluation trajectories of ScanNet.
 - 2025-1-8: demo view selection improved, better quality for multiple rooms.
@@ -86,7 +87,9 @@ some [tips](https://github.com/facebookresearch/mvdust3r/issues/5#issuecomment-2
 
 We use five data for training and test: [ScanNet](https://github.com/ScanNet/ScanNet), [ScanNet++](https://kaldir.vc.in.tum.de/scannetpp/), [HM3D](https://aihabitat.org/datasets/hm3d/), [Gibson](https://github.com/StanfordVL/GibsonEnv/blob/master/gibson/data/README.md), [MP3D](https://niessner.github.io/Matterport/). Please go to their website to sign contract, download and extract them in the folder [data](https://github.com/facebookresearch/mvdust3r/tree/main/data). [Here](https://github.com/facebookresearch/mvdust3r/tree/main/data) are more instructions.
 
-Currently we released the [trajectories](https://huggingface.co/Zhenggang/MV-DUSt3R/tree/main/trajectories) of ScanNet for evaluation. Please download it to the folder [trajectories](https://github.com/facebookresearch/mvdust3r/tree/main/trajectories) More trajectories for training and more data will be released later.
+Currently we released the [trajectories](https://huggingface.co/Zhenggang/MV-DUSt3R/tree/main/trajectories) of ScanNet for evaluation. Please download it to the folder [trajectories](https://github.com/facebookresearch/mvdust3r/tree/main/trajectories).
+
+We also open-sourced the code to generate train and evaluation trajectories for ScanNet and all datasets in Habitat-Sim (HM3D, Gibson, MP3D). please check [here]. (https://github.com/facebookresearch/mvdust3r/tree/main/scripts/tuple_gen)
 
 ## Evaluation
 

data/README.md

@@ -2,8 +2,10 @@
 
 ## Instructions for ScanNet
 
-Scannet is structured as many folders named `scenexxxx_xx`. create a folder `scannet` in this directory and put `scene*` inside it.
+Scannet is structured as folders `scenexxxx_xx`. create a folder `scannet` in this directory and put `scenexxxx_xx` inside it.
 
-## Instructions for other data
+## Instructions for other Habitat-Sim (HM3D, Gibson, MP3D)
 
-TBD...
+HM3D, Gibson, MP3D contains `*.glb` and `*.navmesh` for each scene. put them in `habitat-sim-data/hm3d/train/`, `habitat-sim-data/hm3d/val/`, `habitat-sim-data/gibson/` and `habitat-sim-data/mp3d` for hm3d train, hm3d validation, gibson and mp3d respectively.
+
+Here, we have a file `hs_scene_data.json` as the scene location config.

data/hs_scene_data.json

@@ -22,6 +22,7 @@
     args.tgt_dir = args.data_dir
 
 def extract_scene_name(x, data_name_):
+    print(x, data_name_)
     if "_meta" == data_name_[-5:]:
         data_name = data_name_[:-5]
     else:
@@ -34,14 +35,14 @@ def extract_scene_name(x, data_name_):
         for x_ in xx:
             if 'scene' in x_:
                 return x_
-    elif "habitat_sim" in x:
+    elif "habitat_sim" in x or "habitatSim" in x:
         if "gibson" in x:
             return 'gibson'
         if "mp3d" in x:
             return "mp3d"
         xx = x.split('/')
         for x_ in xx:
-            if x_[5] == "-":
+            if len(x_) >= 6 and x_[5] == "-":
                 return x_
     raise NotImplementedError
 

datasets_preprocess/gather_metadata.py

@@ -0,0 +1 @@
+Requirement already satisfied: numpy in /home/zt15/miniconda3/envs/habitat/lib/python3.9/site-packages (1.24.4)

datasets_preprocess/hs_data_gen/=2.0

@@ -0,0 +1,95 @@
+# Copyright (C) 2022-present Naver Corporation. All rights reserved.
+# Licensed under CC BY-NC-SA 4.0 (non-commercial use only).
+
+"""
+Script to generate image pairs for a given scene reproducing poses provided in a metadata file.
+"""
+import os
+import sys
+sys.path.append("/home/zgtang/data_gen/croco/")
+print(sys.path)
+from datasets.hs.multiview_habitat_sim_generator import MultiviewHabitatSimGenerator
+from datasets.hs.paths import SCENES_DATASET
+import argparse
+import quaternion
+import PIL.Image
+import cv2
+import json
+from tqdm import tqdm
+
+def generate_multiview_images_from_metadata(metadata_filename,
+                                            output_dir,
+                                            overload_params = dict(),
+                                            scene_datasets_paths=None,
+                                            exist_ok=False):   
+    """
+    Generate images from a metadata file for reproducibility purposes.
+    """
+    # Reorder paths by decreasing label length, to avoid collisions when testing if a string by such label
+    if scene_datasets_paths is not None:
+        scene_datasets_paths = dict(sorted(scene_datasets_paths.items(), key= lambda x: len(x[0]), reverse=True))
+
+    with open(metadata_filename, 'r') as f:
+        input_metadata = json.load(f)
+    metadata = dict()
+    for key, value in input_metadata.items():
+        # Optionally replace some paths
+        if key in ("scene_dataset_config_file", "scene", "navmesh") and value != "":
+            if scene_datasets_paths is not None:
+                for dataset_label, dataset_path in scene_datasets_paths.items():
+                    if value.startswith(dataset_label):
+                        value = os.path.normpath(os.path.join(dataset_path, os.path.relpath(value, dataset_label)))
+                        break
+        metadata[key] = value
+
+    # Overload some parameters
+    for key, value in overload_params.items():
+        metadata[key] = value
+
+    generation_entries = dict([(key, value) for key, value in metadata.items() if not (key in ('multiviews', 'output_dir', 'generate_depth'))])
+    generate_depth = metadata["generate_depth"]
+
+    os.makedirs(output_dir, exist_ok=exist_ok)
+ 
+    generator = MultiviewHabitatSimGenerator(**generation_entries)
+
+    # Generate views
+    for idx_label, data in tqdm(metadata['multiviews'].items()):
+        positions = data["positions"]
+        orientations = data["orientations"]
+        n = len(positions)
+        for oidx in range(n):
+            observation = generator.render_viewpoint(positions[oidx], quaternion.from_float_array(orientations[oidx]))
+            observation_label = f"{oidx + 1}" # Leonid is indexing starting from 1
+            # Color image saved using PIL
+            img = PIL.Image.fromarray(observation['color'][:,:,:3])
+            filename = os.path.join(output_dir, f"{idx_label}_{observation_label}.jpeg")
+            img.save(filename)
+            if generate_depth:
+                # Depth image as EXR file
+                filename = os.path.join(output_dir, f"{idx_label}_{observation_label}_depth.exr")
+                cv2.imwrite(filename, observation['depth'], [cv2.IMWRITE_EXR_TYPE, cv2.IMWRITE_EXR_TYPE_HALF])
+                # Camera parameters
+                camera_params = dict([(key, observation[key].tolist()) for key in ("camera_intrinsics", "R_cam2world", "t_cam2world")])
+                filename = os.path.join(output_dir, f"{idx_label}_{observation_label}_camera_params.json")
+                with open(filename, "w") as f:
+                    json.dump(camera_params, f)
+                # Save metadata
+    with open(os.path.join(output_dir, "metadata.json"), "w") as f:
+        json.dump(metadata, f)
+
+    generator.close()
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--metadata_filename", required=True)
+    parser.add_argument("--output_dir", required=True)
+    args = parser.parse_args()
+
+    generate_multiview_images_from_metadata(metadata_filename=args.metadata_filename,
+                             output_dir=args.output_dir,
+                             scene_datasets_paths=SCENES_DATASET,
+                             overload_params=dict(),
+                             exist_ok=True)
+
+ 

datasets_preprocess/hs_data_gen/__init__.py

@@ -0,0 +1,27 @@
+# Copyright (C) 2022-present Naver Corporation. All rights reserved.
+# Licensed under CC BY-NC-SA 4.0 (non-commercial use only).
+
+"""
+Script generating commandlines to generate image pairs from metadata files.
+"""
+import os
+import glob
+from tqdm import tqdm
+import argparse
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--input_dir", required=True)
+    parser.add_argument("--output_dir", required=True)
+    parser.add_argument("--prefix", default="", help="Commanline prefix, useful e.g. to setup environment.")
+    args = parser.parse_args()
+
+    input_metadata_filenames = glob.iglob(f"{args.input_dir}/**/metadata.json", recursive=True)
+
+    for metadata_filename in tqdm(input_metadata_filenames):
+        output_dir = os.path.join(args.output_dir, os.path.relpath(os.path.dirname(metadata_filename), args.input_dir))
+        # Do not process the scene if the metadata file already exists
+        if os.path.exists(os.path.join(output_dir, "metadata.json")):
+            continue
+        commandline = f"{args.prefix}python datasets/habitat_sim/generate_from_metadata.py --metadata_filename={metadata_filename} --output_dir={output_dir}"
+        print(commandline)