Physics-based 3D Avatar Full-Body Motion Control from Sparse Data and Direction Control using Deep Reinforcement Learning

Contributions

The contributions of this work are as follow:

1. Enables a VR user with limited mobility to control a 3D avatar and explore spacious virtual environment
2. Enables object interaction that satisfies physical laws with a key feature of sparse sensor reward

Results and Applications

Visualization result

(Top) Motion sequences of the VR user performing throwing motion with direction control. (Bottom) Reconstructed full-body motion of the 3D avatar.

(Top) Motion sequences of the VR user performing punch motion with direction control. (Bottom) Reconstructed full-body motion of the 3D avatar.

Application

Blocking a projectile

Punching an object

1. Dependencies

• IssacGym Pr4
  
  • Link: Webpage
• Composite Motion Learning with Task Control [Xu et al. SIGGRAPH 2023]
  • Link: Github

2. Motion Data

Motion captured data

• We provide our motion data in assets/motions and assets/retargeted.
  • Motions located in assets/motions are provided by Composite Motion Learning with Task Control.
  • The blocking/pickup/punch/throw motions located in assets/retargeted are extracted from the demo provided by Mixamo.
  • Tennis motions shown in the project is not provided due to the commercial license.
  • Motions extracted from Mixamo are retargeted to our skeleton model with the code provided by ASE. Please refer to an example retargeting script in ase/poselib/retarget_motion.py.

Motion sequence demo provided by Mixamo.
Block, pick fruits, punch motions are illustrated in the rows from top to bottom.

Example demo of tennis motion retargeted to our model

Real VR user motion dataset

Overview of collecting VR user motion data

Data Acquisition from Movement SDK

• Real sensor data of VR headsets and motions of A VR user are extracted using MovementSDK and Unity3D.

Retargeting process

• Same retargeting method used in motion captured data is utilized to transfer the motion of real VR user's motion onto our skeleton model of IsaacGym.

3. Policy Training

python main.py <configure_file> --ckpt <checkpoint_dir>

• Example code of training punching motion
  • options
    • --headless: Run headless without creating a viewer window
    • --server: which server you are running the code
    • Others are stated in main.py

python main.py config/punch_joystick.py --ckpt 1121_punch_MIX --headless --server local

4. Policy Evaluation

python main.py <configure_file> --ckpt <checkpoint_dir> --test

• Example code of testing punching motion

python main.py config/punch_joystick.py --ckpt 1121_punch_MIX/ckpt-20000 --test