发表机构
Tsinghua University; State Key Lab of General Artificial Intelligence, BIGAI; Peking University（清华大学; 通用人工智能国家重点实验室，字节跳动公司; 北京大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究如何通过遥操作赋予机器人手内灵巧性，提出TeleDexter手 - 物体协同跟踪控制器，利用混合奖励训练，经单阶段强化学习及随机化处理可零样本迁移到真实机器人，在多项任务上取得高成功率并能训练自主策略。
AI 中文摘要
人类能够熟练地使用工具、切换抓握方式并在单手中重新定位物体，无缝协调包括平移、重新定向和手指移动的接触转换。通过遥操作赋予机器人灵巧的手这种水平的手内灵巧性，需要通过动态手 - 物体接触精确控制物体运动，但目前的遥操作系统仍远未具备此能力。为弥合这一差距，我们引入了TeleDexter，一种手 - 物体协同跟踪控制器，它将操作员意图映射到学习到的低级接触执行中。该控制器在从人类参考运动导出的连续协同跟踪子目标上进行训练，利用将稀疏子目标目标与密集跟踪奖励相结合的混合奖励，以实现跨多种交互模式的学习。整个管道仅需要单阶段强化学习，并通过随机动作掩码和域随机化，零样本迁移到真实机器人。我们在跨越两个灵巧手的七个具有挑战性的灵巧遥操作任务上评估了TeleDexter，在所有基线均持续失败的情况下，平均成功率达到75%。此外，收集的演示通过行为克隆成功训练了自主策略，朝着人类水平的灵巧遥操作迈出了具体一步。
英文摘要
Humans routinely wield tools, swap grasps, and reposition objects within a single hand, seamlessly orchestrating contact transitions that span translation, reorientation, and finger gaiting. Endowing robot dexterous hands with this level of in-hand dexterity through teleoperation requires precise control of object motion via dynamic hand-object contact, yet current teleoperation systems remain far from this capability. To bridge this gap, we take a major step towards human-level dexterous teleoperation by introducing TeleDexter, a hand-object co-tracking controller that maps operator intent into learned, low-level contact execution. The controller is trained on consecutive co-tracking subgoals derived from human reference motions, utilizing a hybrid reward that couples sparse subgoal objectives with dense tracking rewards to enable learning across diverse interaction modalities rather than frame-wise trajectory imitation. The entire pipeline requires only single-stage RL and, with random action masking and domain randomization, transfers zero-shot to the real robot. We evaluate TeleDexter on seven challenging dexterous teleoperation tasks spanning object reorientation and long-horizon tool use across two dexterous hands, achieving a 75% average success rate where all baselines consistently fail. Furthermore, the collected demonstrations successfully train autonomous policies via behavioral cloning, marking a concrete step towards human-level dexterous teleoperation.
CommentsProject Website: https://bigai-dex.github.io/blog/teledexter/
