通过触觉梦想学习多功能人形操作
Learning Versatile Humanoid Manipulation with Touch Dreaming
AI总结：
本文提出HTD模型，结合触觉、视觉和本体感觉，通过触觉梦想增强学习实现高灵活性的人形操作，实验证明在五个真实任务中成功率提升显著。
摘要：
人形机器人有望实现通用协助，但现实中的全身体能操作仍具挑战性，因其需要全身稳定性、末端执行器灵活性和在频繁接触变化下的接触感知。本文研究了灵活且接触丰富的全身体能操作。我们首先开发了一种基于强化学习的下肢控制器，作为复杂操作中全身执行的稳定性基础。在此控制器基础上，我们开发了一种基于VR的全身人形数据采集系统，集成了灵活的手和触觉传感以实现接触丰富的操作。然后我们提出了带有触觉梦想的人形Transformer（HTD），一种多模态编码器-解码器Transformer，将触觉作为核心模态与多视角视觉和本体感觉相结合。HTD在单阶段训练中通过行为克隆增强触觉梦想：除了预测动作块，策略还预测未来手关节力和未来触觉潜在特征，这些触觉-潜在特征目标由指数移动平均目标编码器提供，无需单独的触觉预训练阶段。这促使策略学习接触感知的表示以实现灵活的操作。在五个真实接触丰富的任务中，HTD在更强的基线基础上平均成功率提升了90.9%。消融结果进一步表明，潜在空间触觉预测比原始触觉预测更有效，使成功率提升30%。这些结果表明，我们的触觉梦想增强学习系统能够实现现实世界中的多功能、高灵活性的人形操作。更多信息和开源材料请访问：humanoid-touch-dream.github.io。
英文摘要：
Humanoid robots promise general-purpose assistance, yet real-world humanoid loco-manipulation remains challenging because it requires whole-body stability, end-effector dexterity, and contact-aware interaction under frequent contact changes. In this work, we study dexterous, contact-rich humanoid loco-manipulation. We first develop an RL-based lower-body controller that serves as the stability backbone for whole-body execution during complex manipulation. Building on this controller, we develop a VR-based whole-body humanoid data collection system that integrates dexterous hands and tactile sensing for contact-rich manipulation. We then propose Humanoid Transformer with Touch Dreaming (HTD), a multimodal encoder-decoder Transformer that models touch as a core modality alongside multi-view vision and proprioception. HTD is trained in a single stage with behavioral cloning augmented by touch dreaming: in addition to predicting action chunks, the policy predicts future hand-joint forces and future tactile latents, with tactile-latent targets provided by an exponential moving average target encoder without requiring a separate tactile pretraining stage. This encourages the policy to learn contact-aware representations for dexterous manipulation. Across five real-world contact-rich tasks, HTD achieves a 90.9% relative improvement in average success rate over the stronger baseline for each task. Ablation results further show that latent-space tactile prediction is more effective than raw tactile prediction, yielding a 30% relative gain in success rate. These results demonstrate that our touch-dreaming-enhanced learning system enables versatile, high-dexterity humanoid manipulation in the real world. More information and open-source materials are available at humanoid-touch-dream.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
