This repository contains a reproducible pipeline for semantic segmentation of dense, colored indoor point clouds using three state-of-the-art models (KPConv, MinkowskiNet, and Sonata). It is submitted as a solution for the R&D Assignment Problem from the Artificial Intelligence for Autonomous Systems Laboratory.
Below is an example of segmentation results from the Sonata model:
|
Sonata Model Results: This visualization demonstrates the semantic segmentation capabilities of the Sonata model on dense indoor point clouds. The model successfully identifies various semantic classes such as walls, floors, furniture, and other objects with high accuracy. Note: The segmentation preserves the original point cloud structure while assigning semantic labels to each point. |
|
This project includes two detailed reports that correspond to the main parts of the R&D assignment:
-
State-of-the-Art Research (
sota_review.md): This report covers Part 1 of the assignment. It includes a comprehensive analysis of SOTA methods, a ranked list of top approaches, and the criteria used for their evaluation. -
Reproduction and Analysis Report (
analysis_report.md): This document addresses Part 2 of the assignment. It details the practical challenges, experiments with data preprocessing, and provides a thorough visual analysis of the segmentation results from the prototyped models.
Before you begin, ensure you have the following installed on your host machine:
- Docker
- NVIDIA Container Toolkit for GPU support
sudo apt update
sudo apt install git unzip htop -y
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh ./get-docker.shcurl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list \
&& sudo apt-get update
sudo apt-get install -y nvidia-container-toolkitConfigure Docker to use the NVIDIA runtime and restart the service.
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart dockerFollow these steps to set up the project environment and run the inference pipeline.
First, clone this repository.
git clone --recursive https://github.com/nikborovets/3d_semseg_rnd_task.git
cd 3d_semseg_rnd_taskUse the provided Makefile to build the Docker container and download the necessary data.
# This command will download data and build the docker image
make -j all
# Alternatively, you can run these steps separately
# make download_data
# make build_dockerRun the Docker container in detached mode.
docker compose up -dEnter the running container to compile C++/CUDA wrappers required by KPConv.
docker exec -it 3d_semseg_rnd_task-pointcept_me-1 /bin/bash
# Inside the container, run the following commands:
cd third_party/KPConv-PyTorch/cpp_wrappers
bash compile_wrappers.sh
cd ../../../To ensure code quality and consistency, this project uses pre-commit hooks. Install them to your Git repository and run them on all files:
uv run pre-commit install
uv run pre-commit run --all-filesOnce the setup is complete, you can run inference using the provided scripts.
Execute the following scripts to run segmentation with the default model configurations.
python mink_inference.py
python sonata_inference.py
python kpconv_inference.pyYou can also run the inference scripts with custom arguments to adjust the models' behavior.
python mink_inference.py \
--voxel_size 0.03
python sonata_inference.py \
--downsampling_method "grid" \
--voxel_size 0.03 \
--enc_patch_size 512
python kpconv_inference.py \
--downsampling_method "grid" \
--voxel_size 0.03 \
--chunk_size 300000
The inference scripts save the segmented point clouds as .ply files in the result_plys/ directory. The results for each model are stored in separate subdirectories:
- KPConv:
result_plys/kpconv_plys/ - MinkowskiNet:
result_plys/minkowski_plys/ - Sonata:
result_plys/sonata_plys/
The output filenames are structured to be descriptive and include the model name, input file, and key inference parameters.
Example Filename Format:
<input_filename>_<model_name>_<parameters>_segmented_seed_<seed_number>.ply
Concrete Example (from KPConv):
down0.01_KPConv_Light_KPFCNN_downsample_grid_voxel0.03m_chunk300000_segmented_seed_42.ply
This section details the computational environment and software used for running experiments and visualizing results.
The resulting .ply point cloud files were visualized and analyzed using MeshLab, an open-source system for processing and editing 3D triangular meshes. It provides a comprehensive set of tools for inspection, cleaning, and rendering large 3D models.
The experiments were conducted on high-performance computing resources from two primary sources:
A significant portion of the computations was performed on the MIPT cluster with the following configuration:
- CPU: 2x Intel(R) Xeon(R) Gold 6136 @ 3.00GHz (total 48 cores, 96 threads)
- RAM: 252 GB
- GPU: 8x NVIDIA GeForce RTX 2080 Ti (11 GB VRAM each)
- CUDA Version: 13.0
For additional computational power and flexibility, dedicated servers from Selectel were utilized. The configurations included:
- CPU: AMD EPYC 7763 64-Core Processor (from 8 to 48 vCPUs)
- RAM: From 32 to 60 GB
- GPU:
- NVIDIA RTX A5000 (24 GB VRAM)
- NVIDIA A100 (40 GB VRAM)
- CUDA Version: 12.2
- PLY File Viewer: For quick online visualization and GIF creation, the ImageToSTL platform was used.