arXiv 2603.222642026-03-23cs.RO

UniDex：一种机器人基础套件，用于从第一人称人类视频中实现通用灵巧手控制

UniDex: A Robot Foundation Suite for Universal Dexterous Hand Control from Egocentric Human Videos

Tsinghua University（清华大学）；Shanghai Qizhi Institute（上海启智研究院）；Sun Yat-sen University（中山大学）；The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）

Gu Zhang, Qicheng Xu, Haozhe Zhang, Jianhan Ma, Long He, Yiming Bao, Zeyu Ping, Zhecheng Yuan, Chenhao Lu, Chengbo Yuan, Tianhai Liang, Xiaoyu Tian, Maanping Sh… 展开作者

Gu Zhang, Qicheng Xu, Haozhe Zhang, Jianhan Ma, Long He, Yiming Bao, Zeyu Ping, Zhecheng Yuan, Chenhao Lu, Chengbo Yuan, Tianhai Liang, Xiaoyu Tian, Maanping Shao, Feihong Zhang, Mingyu Ding, Yang Gao, Hao Zhao, Hang Zhao, Huazhe Xu

备注：: Accepted by CVPR 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出UniDex，通过机器人中心数据集、统一视觉-语言-动作策略和实用人类数据采集设置，实现通用灵巧手控制。UniDex-VLA在两项不同手的挑战性工具使用任务中达到81%的平均任务进度，优于现有VLA基线。

摘要：

灵巧操作仍具挑战性，由于收集真实机器人远程操作数据的成本、手部实现的异质性和控制的高维性。我们提出了UniDex，一种机器人基础套件，结合大规模机器人中心数据集、统一的视觉-语言-动作（VLA）策略和实用的人类数据采集设置，用于通用灵巧手控制。首先，我们构建了UniDex数据集，一个覆盖50000条轨迹的机器人中心数据集，涵盖八种灵巧手（6-24自由度），源自第一人称人类视频数据集。为了将人类数据转换为机器人可执行轨迹，我们采用人机协同的重定向过程对指尖轨迹进行对齐，同时保持合理的手-物体接触。我们操作于显式的3D点云，人类手部被遮挡以缩小运动学和视觉差距。其次，我们引入了功能-执行器对齐空间（FAAS），一种统一的动作空间，将功能相似的执行器映射到共享坐标，实现跨手转移。利用FAAS作为动作参数化，我们训练了UniDex-VLA，一种预训练于UniDex数据集并用任务演示微调的3D VLA策略。此外，我们构建了UniDex-Cap，一种简单的便携式采集设置，记录同步的RGB-D流和人类手部姿态，并将其转换为机器人可执行的轨迹，以实现人机数据共训练，减少对昂贵机器人演示的依赖。在两项不同手的挑战性工具使用任务中，UniDex-VLA实现了81%的平均任务进度，并大幅优于现有VLA基线，同时表现出强大的空间、物体和零样本跨手泛化能力。共同，UniDex数据集、UniDex-VLA和UniDex-Cap为通用灵巧操作提供了可扩展的基础套件。

英文摘要：

Dexterous manipulation remains challenging due to the cost of collecting real-robot teleoperation data, the heterogeneity of hand embodiments, and the high dimensionality of control. We present UniDex, a robot foundation suite that couples a large-scale robot-centric dataset with a unified vision-language-action (VLA) policy and a practical human-data capture setup for universal dexterous hand control. First, we construct UniDex-Dataset, a robot-centric dataset over 50K trajectories across eight dexterous hands (6--24 DoFs), derived from egocentric human video datasets. To transform human data into robot-executable trajectories, we employ a human-in-the-loop retargeting procedure to align fingertip trajectories while preserving plausible hand-object contacts, and we operate on explicit 3D pointclouds with human hands masked to narrow kinematic and visual gaps. Second, we introduce the Function-Actuator-Aligned Space (FAAS), a unified action space that maps functionally similar actuators to shared coordinates, enabling cross-hand transfer. Leveraging FAAS as the action parameterization, we train UniDex-VLA, a 3D VLA policy pretrained on UniDex-Dataset and finetuned with task demonstrations. In addition, we build UniDex-Cap, a simple portable capture setup that records synchronized RGB-D streams and human hand poses and converts them into robot-executable trajectories to enable human-robot data co-training that reduces reliance on costly robot demonstrations. On challenging tool-use tasks across two different hands, UniDex-VLA achieves 81% average task progress and outperforms prior VLA baselines by a large margin, while exhibiting strong spatial, object, and zero-shot cross-hand generalization. Together, UniDex-Dataset, UniDex-VLA, and UniDex-Cap provide a scalable foundation suite for universal dexterous manipulation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。