Authors: Markus Knauer, Samuel Bustamante Gomez, Thomas Eiband, Alin Albu-Schäffer, Freek Stulp, and João Silvério
Responsible: Markus Knauer (markus.knauer@dlr.de) Research Scientist @ German Aerospace Center (DLR), Institute of Robotics and Mechatronics, Munich, Germany & Doctoral candidate & Teaching Assistant @ Technical University of Munich (TUM), Germany.
This repository contains the open-source code for our paper on interactive robot skill adaptation using natural language and tool-based LLM architectures.
If you are interested, you can find similar projects on https://markusknauer.github.io
IEEE paper | ArXiv paper | ELIB paper | YouTube
Ctrl+Click to open links in a new tab
Foundation models have demonstrated impressive capabilities across diverse domains, while imitation learning provides principled methods for robot skill adaptation from limited data. We present a novel framework that enables open-vocabulary skill adaptation through a tool-based architecture maintaining a protective abstraction layer between the language model and robot hardware. Our approach leverages pre-trained LLMs to select and parameterize specific tools for adapting robot skills without requiring fine-tuning or direct model-to-robot interaction. We demonstrate the framework on a 7-DoF torque-controlled robot performing an industrial bearing ring insertion task, showing successful skill adaptation through natural language commands for speed adjustment, trajectory correction, and obstacle avoidance while maintaining safety, transparency, and interpretability.
The system implements a five-step workflow (Algorithm 1 in the paper):
- User Query: User observes robot execution and provides a natural language instruction
- Tool Selection: LLM selects appropriate tool(s) from the available tool set via JSON schema-based function calling
- Tool Parameterization: LLM extracts parameters from the instruction and environment observations
- Tool Execution: Validated tool modifies the KMP trajectory representation
- Feedback: Robot executes modified skill, user provides further instructions
User Command (natural language)
│
▼
┌─────────────────────────┐
│ LLM (tool calling) │ ← OpenAI-compatible API (Ollama, vLLM, OpenAI, ...)
│ Selects & parameterizes│
│ tools from JSON schema │
└────────┬────────────────┘
│
▼
┌─────────────────────────┐
│ Validation & Execution │ ← Type, range, workspace checks
│ • SpeedModulation │ (Paper Section IV-C)
│ • ViaPointInsertion │
│ • ObstacleAvoidance │
└────────┬────────────────┘
│
▼
┌─────────────────────────┐
│ KMP Model │ ← Kernelized Movement Primitives
│ Trajectory update │ (Paper Section III)
└────────┬────────────────┘
│
▼
┌─────────────────────────┐
│ Robot Execution │ ← PyBullet simulation or real robot
└─────────────────────────┘
| Tool | Paper Section | Purpose | Parameters |
|---|---|---|---|
SpeedUpRobot / SlowDownRobot |
IV-D, Eq. 3-4 | Temporal trajectory adjustment | speed_value, adaptation_start, adaptation_end |
AddViaPointsAtTime |
IV-D | Spatial trajectory correction via KMP via-points | input_values (times), output_values (positions) |
AddRepulsionPoint |
IV-D, Eq. 5-6 | Obstacle avoidance via SDF-based correction | position, radius/dimensions |
GetViaPoints / RemoveViaPoints* |
- | Via-point management | - |
GetKMPParameters / SetKMPParameters |
- | KMP hyperparameter adjustment | kernel_length |
TellUserAMessage |
- | Robot-to-user communication | message |
NoToolIsAvailable |
- | Fallback when no tool applies | reason |
irosa/
├── core/
│ ├── tool.py # Tool base class + Toolkit (JSON schema generation)
│ ├── llm_client.py # OpenAI-compatible LLM client with tool calling
│ └── trajectory_corrector.py # SDF-based obstacle avoidance (sphere, OBB)
├── models/
│ ├── kmp_core.py # KMP algorithm (GMM, kernel, via-points)
│ └── kmp.py # KMP wrapper for LLM interaction
├── robots/
│ ├── robot.py # Abstract robot with speed modulation
│ └── sim.py # PyBullet simulation environment
├── tool_definitions/
│ ├── general.py # NoToolIsAvailable, TellUserAMessage
│ ├── kmp.py # Via-point and obstacle avoidance tools
│ └── robot.py # SpeedUpRobot, SlowDownRobot
├── main.py # Interactive loop (Algorithm 1)
└── exceptions.py
The KMP algorithm used in this project is based on our previous work on task-parameterized movement primitives:
- Code: DLR-RM/interactive-incremental-learning
- Paper: M. Knauer et al., "Interactive incremental learning of generalizable skills with local trajectory modulation," IEEE RA-L, 2025.
git clone https://github.com/DLR-RM/IROSA.git
cd IROSA
pip install -e .pip install -e ".[sim]"pip install -e ".[tests,sim]"Each experiment script reproduces one of the three evaluation scenarios from Section V of the paper. The scripts demonstrate the tool-based adaptation workflow without requiring an LLM connection.
Command: "slow down between box and station" — demonstrates trajectory segment determination and speed modulation.
python examples/experiment_speed.py
Command: "Check the ring with the camera on the left" — demonstrates new object integration and via-point insertion.
python examples/experiment_viapoint.py
Command: "Please avoid the blue box" — demonstrates OBB SDF construction, trajectory correction, and repulsion points.
python examples/experiment_obstacle.py
Requires a running LLM server (e.g., Ollama) with a model that supports tool/function calling:
# Start Ollama with a model
ollama pull qwen2.5:72b
# Run IROSA interactive loop
python -m irosa.main --demo data/demonstrations.npz --llm-model qwen2.5:72bSupported LLM backends (any OpenAI-compatible API):
- Ollama:
--llm-url http://localhost:11434/v1 --llm-key ollama - vLLM:
--llm-url http://localhost:8000/v1 --llm-key vllm - OpenAI:
--llm-url https://api.openai.com/v1 --llm-key sk-...
import numpy as np
from irosa.models.kmp import KMPWrapper
# Train KMP from demonstrations
model = KMPWrapper(demonstration_path="data/demonstrations.npz", force_retrain=True)
# Speed modulation: slow down between t=0.55 and t=0.72
model.robot.change_predicting_frequency(
percentage_factor=50, adaptation_start=0.55, adaptation_end=0.72
)
# Via-point insertion: add waypoint at camera position
model.add_viapoints(
input_via=[[0.5]],
output_via=[[0.6, 0.3, 0.25, 1.0, 0.0, 0.0, 0.0]],
)
# Obstacle avoidance: avoid box with OBB dimensions
model.add_repulsion_point(
position=np.array([0.75, 0.35, 0.23]),
dimensions=[0.15, 0.225, 0.235], # width, length, height in meters
safety_margin=0.02, # 2cm safety margin
)make testmake commit-checks # format + type check + lint
make test # run testsIf you use our ideas in a research project or publication, please cite as follows:
@article{knauer2026irosa,
author={Knauer, Markus and Bustamante Gomez, Samuel and Eiband, Thomas and Albu-Sch{\"a}ffer, Alin and Stulp, Freek and Silv{\'e}rio, Jo{\~a}o},
journal={IEEE Robotics and Automation Letters (RA-L)},
title={{IROSA}: Interactive Robot Skill Adaptation Using Natural Language},
year={2026},
volume={11},
number={4},
pages={5246-5253},
doi={10.1109/LRA.2026.3671560},
keywords={LLM; Tool Use; Robot Learning; Movement Primitives; Natural Language},
}If you use the KMP algorithm, please also cite:
@ARTICLE{knauer2025,
author={Knauer, Markus and Albu-Sch{\"a}ffer, Alin and Stulp, Freek and Silv{\'e}rio, Jo{\~a}o},
journal={IEEE Robotics and Automation Letters (RA-L)},
title={Interactive incremental learning of generalizable skills with local trajectory modulation},
year={2025},
volume={10},
number={4},
pages={3398-3405},
doi={10.1109/LRA.2025.3542209}
}
