UniHM: 一体化的视觉语言模型用于统一的灵巧手操作
UniHM: Unified Dexterous Hand Manipulation with Vision Language Model
- ShanghaiTech University（上海科技大学）
- InstAdapt
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
UniHM通过统一的视觉语言模型实现灵巧手操作，利用开放词汇指令提升泛化能力和物理可行性。
AI中文摘要：
规划物理上可行的灵巧手操作是机器人操作和具身AI中的核心挑战。先前的工作通常依赖于以对象为中心的提示或精确的手-物体交互序列，忽略了开放词汇指令中的丰富、组合性指导。我们引入UniHM，第一个由自由形式语言命令引导的统一灵巧手操作框架。我们提出了一种统一的手-灵巧性标记器，将异构的灵巧手形态映射到一个共享的代码本中，提高了跨灵巧手的泛化能力和对新形态的可扩展性。我们的视觉语言动作模型仅在人类-物体交互数据上训练，消除了对大量现实世界远程操作数据集的依赖，并在开放式的语言指令下产生人类般的操作序列方面表现出强大的泛化能力。为了确保物理真实性，我们引入了一个物理引导的动态细化模块，该模块在生成和时间先验下对关节进行分段优化，产生平滑且物理上可行的操作序列。在多个数据集和现实世界评估中，UniHM在已见和未见的对象和轨迹上均取得最先进的结果，展示了强大的泛化能力和高物理可行性。我们的项目页面在https://unihm.github.io/。
英文摘要：
Planning physically feasible dexterous hand manipulation is a central challenge in robotic manipulation and Embodied AI. Prior work typically relies on object-centric cues or precise hand-object interaction sequences, foregoing the rich, compositional guidance of open-vocabulary instruction. We introduce UniHM, the first framework for unified dexterous hand manipulation guided by free-form language commands. We propose a Unified Hand-Dexterous Tokenizer that maps heterogeneous dexterous-hand morphologies into a single shared codebook, improving cross-dexterous hand generalization and scalability to new morphologies. Our vision language action model is trained solely on human-object interaction data, eliminating the need for massive real-world teleoperation datasets, and demonstrates strong generalizability in producing human-like manipulation sequences from open-ended language instructions. To ensure physical realism, we introduce a physics-guided dynamic refinement module that performs segment-wise joint optimization under generative and temporal priors, yielding smooth and physically feasible manipulation sequences. Across multiple datasets and real-world evaluations, UniHM attains state-of-the-art results on both seen and unseen objects and trajectories, demonstrating strong generalization and high physical feasibility. Our project page at \href{https://unihm.github.io/}{https://unihm.github.io/}.
