DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation

Imitation learning from human hand motion data presents a promising avenuefor imbuing robots with human-like dexterity in real-world manipulation tasks.Despite this potential, substantial challenges persist, particularly with theportability of existing hand motion capture (mocap) systems and the difficultyof translating mocap data into effective control policies. To tackle theseissues, we introduce DexCap, a portable hand motion capture system, alongsideDexIL, a novel imitation algorithm for training dexterous robot skills directlyfrom human hand mocap data. DexCap offers precise, occlusion-resistant trackingof wrist and finger motions based on SLAM and electromagnetic field togetherwith 3D observations of the environment. Utilizing this rich dataset, DexILemploys inverse kinematics and point cloud-based imitation learning toreplicate human actions with robot hands. Beyond learning from human motion,DexCap also offers an optional human-in-the-loop correction mechanism to refineand further improve robot performance. Through extensive evaluation across sixdexterous manipulation tasks, our approach not only demonstrates superiorperformance but also showcases the system's capability to effectively learnfrom in-the-wild mocap data, paving the way for future data collection methodsfor dexterous manipulation. More details can be found athttps://dex-cap.github.io