PRP-RoboPiano | 双臂机器人钢琴演奏系统

🎹 English

Overview

PRP-RoboPiano is an autonomous dual-arm robotic piano playing system based on JAKA K1 dual-arm robot + OmniHand O10 dexterous hands. The system implements a complete pipeline from MIDI parsing to physical piano performance, featuring intelligent fingering assignment, collision-free trajectory planning, and real-time motion control.

Development Approach: Script-based trajectory planning (Solution A), targeting end-to-end pipeline completion within 2-4 weeks.

🎯 Key Features

Two-Point Calibration: Fast calibration method using only 2 reference points (C2 + C5) to compute all 88 key positions with ±3mm accuracy
Intelligent Fingering: Dynamic programming-based fingering assignment optimized for dual-hand coordination
Three-Layer Trajectory Planning:
- Layer 1: Arm coarse positioning (5-7 key coverage)
- Layer 2: Optimal finger selection
- Layer 3: Fine finger adjustment (±18mm lateral sway)
Finger Kinematics: 5-stage motion model (hover → strike → press → release → return) with black/white key adaptation
Real-time Control: CAN-FD bus integration for JAKA arms and OmniHand dexterous hands
Collision Detection: Real-time dual-arm collision avoidance

📊 Development Status

Module	Path	Status
Module 0 - Calibration	`piano_robot/calibration/`	✅ Complete (12-point + 2-point methods)
Module 1 - MIDI Parsing	`piano_robot/midi_utils/`	✅ Complete
Module 2 - Fingering Assignment	`piano_robot/fingering/`	✅ Complete
Module 3 - Trajectory Planning	`piano_robot/planning/`	✅ Complete (3-layer architecture)
Module 4 - Motion Control	`piano_robot/control/`	✅ Complete (with finger kinematics)
Module 5 - Evaluation	`piano_robot/evaluation/`	⬜ TODO (F1 metrics + MIDI playback)

🏗️ System Architecture

PRP-RoboPiano/
├── piano_robot/                    # Core Python package
│   ├── calibration/                  ← Module 0: Two-point calibration
│   ├── midi_utils/                   ← Module 1: MIDI parsing & Piano Roll
│   ├── fingering/                    ← Module 2: Fingering assignment
│   ├── planning/                     ← Module 3: Trajectory planning
│   ├── control/                      ← Module 4: Motion control & drivers
│   └── evaluation/                   ← Module 5: F1 evaluation
│
├── configs/                        # YAML configuration files
│   ├── piano.yaml                    Piano geometry + calibration results
│   ├── robot.yaml                    JAKA + OmniHand parameters
│   └── runtime.yaml                  Control frequency, safety limits
│
├── data/
│   ├── midi_library/               # 🎵 Place your MIDI files here
│   │   ├── easy/                     Beginner pieces
│   │   ├── medium/                   Intermediate pieces
│   │   ├── hard/                     Advanced pieces
│   │   └── README.md                 Recommended pieces & sources
│   ├── processed/                    Module 1 output (auto-generated)
│   └── calibration/                  Calibration results
│
├── scripts/                        # Stage-by-stage execution scripts
│   ├── 01_calibrate_two_point.py     One-time calibration
│   ├── 02_test_calibration.py        Calibration verification
│   ├── 03_play_with_trajectory.py    🎹 End-to-end performance
│   ├── hardware_tests/               Hardware validation scripts
│   └── make_sample_midi.py           Generate Twinkle Twinkle MIDI
│
├── tests/                          # Unit tests
├── external_sdks/                  # Third-party SDKs
│   ├── jaka_k1_python_binding/       JAKA official binding
│   └── Omnihand-2025-SDK/            OmniHand CAN-FD driver
├── logs/                           # Logs (auto-generated)
└── requirements.txt

🔧 Hardware Requirements

Robot Arms: JAKA K1 dual-arm collaborative robot (7-DOF × 2)
End Effectors: OmniHand O10 dexterous hands (10 active DOF × 2)
Computing Platform: NVIDIA Jetson Orin (ARM64 architecture)
Communication: CAN-FD bus (via Socket CAN)
Piano: Standard 88-key piano (MIDI 21-108)

📦 Installation

1. Clone Repository

git clone https://github.com/Xuliwen-0x01/tests.git
cd PRP-RoboPiano

2. Install Python Dependencies

pip install -r requirements.txt

3. Install Hardware SDKs

JAKA K1 Python Binding:

# Follow official documentation
# https://github.com/JAKARobotics/jaka_k1_python_binding
cd external_sdks/jaka_k1_python_binding
mkdir build && cd build
cmake ..
make -j$(nproc)
sudo make install

OmniHand O10 SDK:

# Follow official documentation
# https://github.com/AgibotTech/Omnihand-2025-SDK
cd external_sdks/Omnihand-2025-SDK
mkdir build && cd build
cmake ..
make -j$(nproc)
sudo make install

4. Configure CAN Bus (on Jetson Orin)

# Enable CAN-FD interface
sudo ip link set can0 type can bitrate 1000000 sample-point 0.8 \
     dbitrate 5000000 dsample-point 0.8 fd on
sudo ifconfig can0 up

# Verify
ip -details link show can0

🚀 Quick Start

Step 1: Generate Sample MIDI

python scripts/make_sample_midi.py
# Generates: data/midi_library/easy/twinkle_twinkle.mid

Step 2: Two-Point Calibration

python scripts/01_calibrate_two_point.py
# Calibrate only 2 points:
#   1. Left index finger → C2 (MIDI 36)
#   2. Right index finger → C5 (MIDI 72)
# System automatically computes all 88 key positions
# Results saved to: configs/piano.yaml

Step 3: Verify Calibration

python scripts/02_test_calibration.py
# Tests middle keys (C3/C4/E4/G4, etc.)
# Generates accuracy report (mean error, max error)
# Target: Mean error < 3mm

Step 4: Hardware Tests

# Single key test
python scripts/hardware_tests/test_single_key.py

# C major scale test
python scripts/hardware_tests/test_scale.py

# Finger sway test
python scripts/hardware_tests/test_finger_sway.py

Step 5: End-to-End Performance

python scripts/03_play_with_trajectory.py data/midi_library/easy/twinkle_twinkle.mid

# Complete pipeline:
#   1. Load calibration results
#   2. Parse MIDI file
#   3. Assign fingering
#   4. Plan trajectories (3-layer architecture)
#   5. Execute performance

🎼 MIDI Library

Download MIDI files from:

MuseScore: musescore.com (Popular/Classical)
MIDI World: midiworld.com (Classics)
PIG Dataset: Piano Fingering Dataset (With professional fingering annotations, highly recommended)

Place downloaded files in data/midi_library/easy/, medium/, or hard/.

🔬 Technical Details

Two-Point Calibration Method

Principle: Leverages standard key spacing (23mm) to compute all key positions from 2 reference points
Reference Points: Left index C2 + Right index C5
Advantage: Reduces calibration time from 12 points to 2 points, maintains ±3mm accuracy

Three-Layer Trajectory Planning

Layer 1 - Arm Coarse Positioning: Select hand center position covering 5-7 keys
Layer 2 - Finger Selection: Choose optimal finger (thumb/index/middle/ring/pinky) based on pitch
Layer 3 - Finger Fine Adjustment: ±18mm lateral sway to extend reach

Finger Physical Model

Sway Capability: Index/ring/pinky ±18mm, middle finger no sway
Thumb Special Handling: Fixed posture, Z-axis +10mm, skip-key playing
Black Key Adjustment: Z-axis +5mm, X-axis +30mm

Finger Kinematics Mapping

5 Fingers: Independent motor position mapping (0-4096 for PIP, 0-2048 for ABAD)
5 Key Press Stages: hover(500) → strike(800) → press(1200) → release(800) → return(500)
Black/White Key Difference: Black key motor position +100 (more curved posture)
Lateral Sway Compensation: Achieved via wrist and metacarpal motors for ±18mm range

📝 Known Issues

See Todo.md for detailed issue tracking.

Critical Issues:

OmniHand joint position ranges vary by joint type (PIP: 0-4096, ABAD: 0-2048)
Requires hardware testing on Jetson Orin platform
Dual-arm parallel control implemented but not yet utilized in motion executor

🤝 Contributing

Contributions are welcome! Please follow these guidelines:

Fork the repository
Create a feature branch (git checkout -b feature/AmazingFeature)
Commit your changes (git commit -m 'Add some AmazingFeature')
Push to the branch (git push origin feature/AmazingFeature)
Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

📧 Contact

Project Lead: Xuliwen
GitHub: @Xuliwen-0x01
Repository: https://github.com/Xuliwen-0x01/tests

🙏 Acknowledgments

JAKA Robotics for the K1 dual-arm robot platform
Agibot (智元机器人) for the OmniHand O10 dexterous hand SDK
RoboPianist (CoRL 2023) for inspiration: google-research/robopianist
Open-source MIDI processing libraries
AI assistance provided by Anthropic Claude

🎹 中文

项目简介

PRP-RoboPiano 是一个基于 JAKA K1 双臂机器人 + 智元 OmniHand O10 灵巧手 的自主钢琴演奏系统。系统实现了从 MIDI 解析到物理钢琴演奏的完整流程，包含智能指法分配、无碰撞轨迹规划和实时运动控制。

开发路线：脚本化轨迹规划（方案 A），目标在 2-4 周内完成端到端流程。

🎯 核心功能

两点标定法：仅需标定 2 个参考点（C2 + C5）即可计算所有 88 个琴键位置，精度 ±3mm
智能指法分配：基于动态规划的指法优化，针对双手协调进行优化
三层轨迹规划：
- 第一层：机械臂粗定位（覆盖 5-7 个琴键）
- 第二层：最优手指选择
- 第三层：手指微调（±18mm 横向摆动）
手指运动学：5 阶段运动模型（悬停 → 击键 → 按压 →释放 → 返回），适配黑白键
实时控制：通过 CAN-FD 总线集成 JAKA 机械臂和 OmniHand 灵巧手
碰撞检测：实时双臂碰撞避免

📊 开发状态

模块	路径	状态
模块 0 - 标定	`piano_robot/calibration/`	✅ 已完成（12点法 + 两点法）
模块 1 - MIDI 解析	`piano_robot/midi_utils/`	✅ 已完成
模块 2 - 指法分配	`piano_robot/fingering/`	✅ 已完成
模块 3 - 轨迹规划	`piano_robot/planning/`	✅ 已完成（三层架构）
模块 4 - 运动控制	`piano_robot/control/`	✅ 已完成（含手指运动学）
模块 5 - 评估	`piano_robot/evaluation/`	⬜ 待完成（F1 指标 + MIDI 回读）

🏗️ 系统架构

PRP-RoboPiano/
├── piano_robot/                    # 核心 Python 包
│   ├── calibration/                  ← 模块 0：两点标定
│   ├── midi_utils/                   ← 模块 1：MIDI 解析 & Piano Roll
│   ├── fingering/                    ← 模块 2：指法分配
│   ├── planning/                     ← 模块 3：轨迹规划
│   ├── control/                      ← 模块 4：运动控制与驱动
│   └── evaluation/                   ← 模块 5：F1 评估
│
├── configs/                        # YAML 配置文件
│   ├── piano.yaml                    钢琴几何 + 标定结果
│   ├── robot.yaml                    JAKA + OmniHand 参数
│   └── runtime.yaml                  控制频率、安全限位
│
├── data/
│   ├── midi_library/               # 🎵 把你的 MIDI 文件放这里
│   │   ├── easy/                     入门曲目
│   │   ├── medium/                   中级曲目
│   │   ├── hard/                     高级曲目
│   │   └── README.md                 推荐曲目和下载来源
│   ├── processed/                    模块 1 输出（自动生成）
│   └── calibration/                  标定结果
│
├── scripts/                        # 分阶段执行脚本
│   ├── 01_calibrate_two_point.py     一次性标定
│   ├── 02_test_calibration.py        标定验证
│   ├── 03_play_with_trajectory.py    🎹 端到端演奏
│   ├── hardware_tests/               硬件验证脚本
│   └── make_sample_midi.py           生成小星星 MIDI
│
├── tests/                          # 单元测试
├── external_sdks/                  # 第三方 SDK
│   ├── jaka_k1_python_binding/       JAKA 官方绑定
│   └── Omnihand-2025-SDK/            OmniHand CAN-FD 驱动
├── logs/                           # 日志（自动生成）
└── requirements.txt

🔧 硬件要求

机械臂：JAKA K1 双臂协作机器人（7-DOF × 2）
末端执行器：OmniHand O10 灵巧手（10 主动自由度 × 2）
计算平台：NVIDIA Jetson Orin（ARM64 架构）
通信接口：CAN-FD 总线（通过 Socket CAN）
钢琴：标准 88 键钢琴（MIDI 21-108）

📦 安装步骤

1. 克隆仓库

git clone https://github.com/Xuliwen-0x01/tests.git
cd PRP-RoboPiano

2. 安装 Python 依赖

pip install -r requirements.txt

3. 安装硬件 SDK

JAKA K1 Python 绑定：

# 按照官方文档操作
# https://github.com/JAKARobotics/jaka_k1_python_binding
cd external_sdks/jaka_k1_python_binding
mkdir build && cd build
cmake ..
make -j$(nproc)
sudo make install

OmniHand O10 SDK：

# 按照官方文档操作
# https://github.com/AgibotTech/Omnihand-2025-SDK
cd external_sdks/Omnihand-2025-SDK
mkdir build && cd build
cmake ..
make -j$(nproc)
sudo make install

4. 配置 CAN 总线（在 Jetson Orin 上）

# 启用 CAN-FD 接口
sudo ip link set can0 type can bitrate 1000000 sample-point 0.8 \
     dbitrate 5000000 dsample-point 0.8 fd on
sudo ifconfig can0 up

# 验证
ip -details link show can0

🚀 快速开始

步骤 1：生成示例 MIDI

python scripts/make_sample_midi.py
# 生成文件：data/midi_library/easy/twinkle_twinkle.mid

步骤 2：两点标定

python scripts/01_calibrate_two_point.py
# 只需标定 2 个点：
#   1. 左手食指 → C2 (MIDI 36)
#   2. 右手食指 → C5 (MIDI 72)
# 系统自动计算所有 88 个琴键位置
# 结果保存到：configs/piano.yaml

步骤 3：验证标定精度

python scripts/02_test_calibration.py
# 测试中间琴键（C3/C4/E4/G4 等）
# 生成精度报告（平均误差、最大误差）
# 目标：平均误差 < 3mm

步骤 4：硬件测试

# 单键测试
python scripts/hardware_tests/test_single_key.py

# C 大调音阶测试
python scripts/hardware_tests/test_scale.py

# 手指摆动测试
python scripts/hardware_tests/test_finger_sway.py

步骤 5：端到端演奏

python scripts/03_play_with_trajectory.py data/midi_library/easy/twinkle_twinkle.mid

# 完整流程：
#   1. 加载标定结果
#   2. 解析 MIDI 文件
#   3. 分配指法
#   4. 规划轨迹（三层架构）
#   5. 执行演奏

🎼 MIDI 音乐库

从以下网站下载 MIDI 文件：

MuseScore：musescore.com（流行/古典）
MIDI World：midiworld.com（经典曲目）
PIG 数据集：钢琴指法数据集（带专业指法标注，强烈推荐）

将下载的文件放入 data/midi_library/easy/、medium/ 或 hard/。

🔬 技术细节

两点标定法

原理：利用标准琴键间距（23mm）从 2 个参考点推算所有琴键位置
参考点：左手食指 C2 + 右手食指 C5
优势：标定时间从 12 点缩短到 2 点，精度保持在 ±3mm 以内

三层轨迹规划

第一层 - 机械臂粗定位：选择手位中心，覆盖 5-7 个琴键
第二层 - 手指选择：根据音高选择最优手指（拇指/食指/中指/无名指/小指）
第三层 - 手指微调：±18mm 横向摆动扩展触及范围

手指物理模型

摆动能力：食指/无名指/小指 ±18mm，中指不摆动
大拇指特殊处理：固定姿态、Z 轴抬高 10mm、隔键弹奏
黑键调整：Z 轴 +5mm、X 轴 +30mm

手指运动学映射

5 个手指：独立的电机位置映射（PIP 关节 0-4096，ABAD 关节 0-2048）
5 个按键阶段：hover(500) → strike(800) → press(1200) → release(800) → return(500)
黑白键差异：黑键电机位置 +100（更弯曲的姿态）
横向摆动补偿：通过手腕和掌骨电机实现 ±18mm 范围

📝 已知问题

详见 Todo.md 获取详细问题跟踪。

关键问题：

OmniHand 关节位置范围因关节类型而异（PIP：0-4096，ABAD：0-2048）
需要在 Jetson Orin 平台上进行硬件测试
双臂并行控制已实现但尚未在运动执行器中使用

🤝 贡献指南

欢迎贡献！请遵循以下准则：

Fork 仓库
创建功能分支（git checkout -b feature/AmazingFeature）
提交更改（git commit -m 'Add some AmazingFeature'）
推送到分支（git push origin feature/AmazingFeature）
开启 Pull Request

📄 许可证

本项目采用 MIT 许可证 - 详见 LICENSE 文件。

📧 联系方式

项目负责人：Xuliwen
GitHub：@Xuliwen-0x01
仓库地址：https://github.com/Xuliwen-0x01/tests

🙏 致谢

JAKA Robotics（节卡机器人） 提供 K1 双臂机器人平台
Agibot（智元机器人） 提供 OmniHand O10 灵巧手 SDK
RoboPianist（CoRL 2023）提供灵感：google-research/robopianist
开源 MIDI 处理库
Anthropic Claude 提供 AI 辅助开发

📚 References | 参考文献

RoboPianist (CoRL 2023): github.com/google-research/robopianist
JAKA Python SDK: jaka.com/docs
OmniHand O10 SDK: github.com/AgibotTech/Omnihand-2025-SDK
PIG Fingering Dataset: beam.kisarazu.ac.jp

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
.claude		.claude
ai_docs		ai_docs
configs		configs
data		data
docs		docs
experiments		experiments
external_sdks/jaka_k1_python_binding		external_sdks/jaka_k1_python_binding
piano_robot		piano_robot
scripts		scripts
tests		tests
.gitignore		.gitignore
PROJECT_CONTEXT.md		PROJECT_CONTEXT.md
README.md		README.md
Todo.md		Todo.md
cleanup_project.py		cleanup_project.py
main.py		main.py
requirements.txt		requirements.txt
test_modules.py		test_modules.py

Folders and files

Latest commit

History

Repository files navigation

PRP-RoboPiano | 双臂机器人钢琴演奏系统

🎹 English

Overview

🎯 Key Features

📊 Development Status

🏗️ System Architecture

🔧 Hardware Requirements

📦 Installation

1. Clone Repository

2. Install Python Dependencies

3. Install Hardware SDKs

4. Configure CAN Bus (on Jetson Orin)

🚀 Quick Start

Step 1: Generate Sample MIDI

Step 2: Two-Point Calibration

Step 3: Verify Calibration

Step 4: Hardware Tests

Step 5: End-to-End Performance

🎼 MIDI Library

🔬 Technical Details

Two-Point Calibration Method

Three-Layer Trajectory Planning

Finger Physical Model

Finger Kinematics Mapping

📝 Known Issues

🤝 Contributing

📄 License

📧 Contact

🙏 Acknowledgments

🎹 中文

项目简介

🎯 核心功能

📊 开发状态

🏗️ 系统架构

🔧 硬件要求

📦 安装步骤

1. 克隆仓库

2. 安装 Python 依赖

3. 安装硬件 SDK

4. 配置 CAN 总线（在 Jetson Orin 上）

🚀 快速开始

步骤 1：生成示例 MIDI

步骤 2：两点标定

步骤 3：验证标定精度

步骤 4：硬件测试

步骤 5：端到端演奏

🎼 MIDI 音乐库

🔬 技术细节

两点标定法

三层轨迹规划

手指物理模型

手指运动学映射

📝 已知问题

🤝 贡献指南

📄 许可证

📧 联系方式

🙏 致谢

📚 References | 参考文献

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages