Visual Grounding as intermediate CoT besides VSI

[Jimntu]

Hi, great work! I have a few questions I would like to ask.

From your paper, my understanding is that you first construct a GCoT dataset based on the VSI benchmark, and then train GS-Reasoner on this GCoT data. During inference, GS-Reasoner grounds the relevant objects and then performs subsequent reasoning.

I would like to check whether the same GCoT process is also applied to other 3D reasoning tasks such as SQA3D or Scan2Cap. The paper says "We attribute these gains to explicitly predicting coordinates for 3D visual grounding, which forces the model to better capture geometric and positional cues, thereby improving dense captioning performance." This made me wonder: for tasks like Scan2Cap, did you construct a GCoT-style dataset (e.g., GCoT-Scan2Cap) to train the model on grounding-augmented reasoning? Or does GS-Reasoner automatically output and leverage coordinate predictions for these tasks even without building a separate GCoT dataset beyond VSI?

Thank you very much, and I look forward to your reply!

[Jimntu]

Hi, I noticed that the bounding-box values and room-size values in the GCoT dataset differ from those in the original VLM-3R metadata. Is this difference caused by the alignment or coordinate transformation used during GCoT generation?

If so, would it be possible for you to release the metadata used for the GCoT dataset as well?

[yiming-cc]

Hi @Jimntu, sorry for the late reply, and thank you for your thoughtful questions. We reply them as follows:

• Regarding the use of GCoT in different tasks:
  In our current implementation, the GCoT construction and training process is applied only to VSIBench. For other 3D reasoning tasks (e.g., SQA3D and Scan2Cap), we explicitly prompt the model to skip the intermediate thinking process and directly output the final answer.
  The performance gains observed on the dense captioning task (Scan2Cap) mainly stem from the visual grounding strategy we adopt—namely, requiring the model to directly predict 3D bounding box coordinates. We believe that enforcing coordinate prediction without relying on any external grounding modules encourages the LLM to internally capture geometric and positional cues more effectively. This enhanced understanding of 3D spatial information naturally transfers to improved dense captioning performance, even without constructing a separate GCoT-style dataset (e.g., GCoT-Scan2Cap).
• Regarding differences in bounding-box and room-size values:
  Yes, this discrepancy is caused by additional axis alignment applied during metadata generation. Specifically, we perform axis alignment for ScanNet++ and ARKitScenes point clouds, as their original coordinate systems are not well aligned. The process involves first aligning the point cloud, obtaining the axis-alignment transformation matrix, and then applying this matrix to all bounding boxes during metadata generation.
  We recommend reproducing this process yourself rather than directly using our metadata, as our metadata is not complete (we filter out certain objects from the raw datasets). In addition, the released GCoT dataset includes data augmentation designed to support autoregressive grounding during training which will also cause bbox and room-size values change. More details about the augmentation strategy can be found in the paper.

[Jimntu]

Thank you so much for your detailed explanation.

However, I have tried to get the metadata of processed point cloud following axis_align_pcd.py, the result is still different from GCoT after considering augmentation. Is it possible for you to provide the script of getting metadata on the processed point cloud if the files are not completed to give us some hints?

Thank you so much!

[Jimntu]

I suspect that the discrepancy might come from a difference in how the transformation matrix is computed in the preprocessing pipelines. May I ask whether you enable --fit_ground in axis_align_pcd.py?

In the paper, it is mentioned that both the ground and walls are aligned. However, in the README comment for axis_align_pcd, there is no --fit_ground (it sets as False). I am a bit unsure which setting is actually used in the released preprocessing, and clarification on this point would be very helpful.

Thank you very much for your time and help.

[yiming-cc]

Hi @Jimntu, When performing axis alignment on the point clouds, we set --fit_ground to false, since the raw point clouds in the original datasets are typically already aligned with the ground plane (i.e., the z-axis is consistent with the negative gravity direction).

Based on our experience, the issue is more likely coming from the metadata generation script on your side. We carefully tested the full preprocessing pipeline before releasing the data. During training, the point clouds (after axis alignment and data augmentation) are consistent with the bounding box annotations provided in the released GCoT dataset. You can verify this by visualizing the aligned point cloud together with the bounding boxes—adding a simple visualization step in the dataset loader is usually the easiest way to debug this.

In our view, the primary source of the discrepancy lies in the bounding box representation. In the original datasets, object annotations are provided as oriented bounding boxes (OBB), while in our pipeline we deliberately convert them to axis-aligned bounding boxes (AABB) to simplify the reasoning process. We believe this OBB-to-AABB conversion is the main factor that leads to the observed differences in bounding box parameters.

For ScanNet, in particular, we derive AABBs directly from the object point clouds using the following procedure:

def get_object_boxes_aabb(points):
    # Create a point cloud
    pcd = o3d.geometry.PointCloud()
    pcd.points = o3d.utility.Vector3dVector(points.astype(np.float64))

    bbox = pcd.get_axis_aligned_bounding_box()
    
    return bbox

We did not release this part of the code, as it is only required during GCoT generation and is not needed for training GS-Reasoner on the released GCoT dataset itself. Moreover, the metadata generation process is highly adaptable and may vary depending on different cases.

[Jimntu]

Thank you very much for your suggestions on verifying the bounding boxes by visualizing them together with the aligned point clouds. Following your advice, I visualized the point clouds after preprocessing and augmentation using the provided metadata, and overlaid them with the bounding boxes from GCOT.

I examined six cases in total and found that two of them may have potential issues. For comparison, I also visualized the same point clouds with bounding boxes generated by my own pipeline (using the same preprocessing and augmentation steps). The resulting figures are shown below.

arkitscene_47115180 "id": "25924322-4256-48a5-85f4-9791f80cbd3b" (gcot_route_plan_train.json)
(left: bbox in gcot with augmented point cloud, right bbox generated by augment_point_cloud_torch with augmented point cloud)

arkitscene_45662813 "id": "ea4de5dd-45b0-4849-9255-c905ba697a28" (gcot_route_plan_train.json)
(left: bbox in gcot with augmented point cloud, right bbox generated by augment_point_cloud_torch with augmented point cloud)

Would it be possible for you to double-check the bounding box information for these two scenes? It is also possible that other scenes may have similar issues, although I have not yet examined them in detail.

Thank you very much for your time and effort!

[Jimntu]

May I ask how frame-level metadata is obtained for ScanNet++ and ARKitScenes, particularly metadata that records the first appearance of objects?

For ScanNet, VLM-3R provides frame metadata that includes object appearance information, which can be derived from the available 2D instance masks. However, for ScanNet++, there does not seem to be any released frame metadata containing object first-appearance information, and to my knowledge there is also no official script for generating frame metadata for ARKitScenes.

Any guidance or recommended practices would be greatly appreciated. Thank you very much for your time and help.