Intent2Motion

Intent2Motion is a voice-controlled robotic manipulation system built using ROS2, local LLM planning, retrieval-based skill routing, dialogue management, and robot arm execution.

Demo Video

The system converts natural language commands such as:

• "pick up the red cube"
• "place it on the shelf"
• "wave hello"
• "go home"

into executable robot actions.

The project supports:

• Voice input
• Local speech-to-text transcription
• Local LLM planning using Ollama
• Dialogue-based clarification
• Pronoun resolution ("pick it up", "place it there")
• Object grounding
• Region grounding
• Robot arm execution in Gazebo
• Text-to-speech responses

---

Features

Voice Assistant

Speak naturally to the robot.

Examples:

• pick up the red cube
• place it on the shelf
• wave hello
• go home

---

Local LLM Planning

Intent2Motion uses a local Ollama model for semantic task planning.

Current model:

• qwen2.5:3b

No cloud API is required for normal operation.

---

Dialogue Management

The system can ask follow-up questions when information is missing.

Example:

User:

pick it up

Robot:

What object would you like me to pick up?

User:

red cube

Robot:

Picked up red cube successfully.

---

Pronoun Resolution

The robot remembers recently manipulated objects.

Example:

User:

pick red cube

User:

place it on shelf

The robot automatically resolves:

it → red cube

---

Object Grounding

Commands are validated against visible objects.

Example:

User:

pick pizza

Robot:

Could not find object: pizza

---

Region Grounding

Commands are validated against known regions.

Supported regions:

• shelf
• desk
• table

---

Holding Conflict Resolution

If the robot is already holding an object:

User:

pick blue block

Robot:

I am already holding red cube. Where should I place it first?

---

System Architecture

User Speech ↓ Whisper STT ↓ Intent Pipeline ↓ RAG Router ↓ Local Planner (Qwen 2.5 3B) ↓ Plan Validation ↓ Dialogue Manager ↓ Plan Executor ↓ ROS2 Interface ↓ Robot Arm

---

Requirements

Operating System

Recommended:

• Ubuntu 22.04

Supported:

• Ubuntu 22.04
• Windows 11 + WSL2 Ubuntu 22.04

---

Python

Python 3.10

---

ROS

ROS2 Humble

---

Ollama

Required for local planning.

---

Linux Installation

1. Clone Repository

git clone https://github.com/octorohan/intent2Motion

cd robarm_llm

---

2. Create Environment

conda create -n robarm python=3.10 -y

conda activate robarm

---

3. Install Python Dependencies

pip install -r requirements.txt

---

4. Install Ollama

Follow official Ollama installation instructions.

After installation:

ollama pull qwen2.5:3b

Verify:

ollama list

Expected:

qwen2.5:3b

---

5. Install ROS2 Humble

Install ROS2 Humble before running the simulator.

Verify:

ros2 --version

---

Windows Installation (WSL2)

1. Install WSL2

Install Ubuntu 22.04 through WSL2.

Verify:

wsl

---

2. Install ROS2 Humble Inside WSL

Follow ROS2 Humble installation instructions for Ubuntu 22.04.

---

3. Install Miniconda

Inside WSL:

Install Miniconda.

Create environment:

conda create -n robarm python=3.10 -y

conda activate robarm

---

4. Install Dependencies

pip install -r requirements.txt

---

5. Install Ollama

Install Ollama.

Download model:

ollama pull qwen2.5:3b

---

6. Verify Installation

ollama list

ros2 --version

python --version

---

Running the Simulator

Open Terminal 1:

ros2 launch robarm_sim robarm_sim.launch.py

Wait approximately 15 seconds.

Open Terminal 2:

ros2 control list_controllers

Expected controllers:

joint_state_broadcaster

arm_controller

---

Running the Voice Assistant

Open a new terminal:

conda activate robarm

cd ~/ros2_ws/src/robarm_llm

python voice/voice_loop.py

---

Voice Assistant Commands

During runtime:

Press ENTER

Start recording.

Press ENTER again

Stop recording.

You can also directly type commands.

Examples:

pick red cube

pick blue block

place it on shelf

wave

go home

---

Built-in Commands

status

Shows current dialogue state.

cancel

Clears current pending dialogue and context.

help

Shows available commands.

q

Quit the application.

---

Example Session

pick red cube

Robot:

Picked up red cube successfully.

place it on shelf

Robot:

Placed object at shelf.

pick blue block

Robot:

Picked up blue block successfully.

wave

Robot:

Waved successfully.

---

Troubleshooting

Ollama Not Running

Start Ollama:

ollama serve

Verify:

ollama list

---

Model Missing

Download:

ollama pull qwen2.5:3b

---

ROS Controllers Missing

Verify:

ros2 control list_controllers

Restart simulator if controllers are not loaded.

---

Voice Transcription Problems

Check microphone permissions.

Try typed commands first.

Verify Whisper dependencies installed correctly.

---

Robot Does Not Move

Verify:

• Gazebo is running
• Controllers are loaded
• ROS topics are available

---

Project Structure

robarm_llm/

├── robarm_llm/

├── planning/

├── skills/

├── rag/

├── dialogue/

├── validation/

├── voice/

├── perception/

├── requirements.txt

└── README.md

---

Future Improvements

• Better ambiguity handling
• Improved dialogue memory
• Multi-object manipulation
• Dynamic perception integration
• Advanced task planning
• Additional robot skills

---

Support

If you get stuck:

1. Read the Troubleshooting section.
2. Ask ChatGPT, Claude, Gemini, or another LLM.
3. Open a GitHub issue.

Most setup issues are related to:

• ROS2 installation
• Conda environment activation
• Ollama setup
• Missing voice models
• Controller startup order

Good luck and have fun building with Intent2Motion.