OpenHOI: 基于多模态大语言模型的开放世界手-物体交互合成
OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Model
AI总结：
OpenHOI通过多模态大语言模型实现开放世界手-物体交互合成，能生成长周期操控序列并处理复杂语言指令。
摘要：
理解并合成逼真的3D手-物体交互（HOI）对于从沉浸式AR/VR到灵巧机器人等应用至关重要。现有方法在泛化能力上存在不足，表现良好于封闭集对象和预定义任务，但无法处理未见对象或开放词汇指令。我们引入OpenHOI，首个开放世界HOI合成框架，能够为新对象生成长周期操控序列，通过自由形式语言指令引导。我们的方法整合了一个针对联合感知接地和语义任务分解进行微调的3D多模态大语言模型（MLLM），能够精确定位交互区域（例如把手、按钮）并将复杂指令（例如“找到一个水瓶并喝一口”）分解为可执行子任务。为了合成物理上合理的交互，我们提出了一种基于感知的扩散模型，配以无需训练的物理修正阶段，以最小化穿透并优化感知对齐。在多样化的场景评估中，OpenHOI在泛化到新对象类别、多阶段任务和复杂语言指令方面均优于最先进的方法。我们的项目页面位于https://openhoi.github.io
英文摘要：
Understanding and synthesizing realistic 3D hand-object interactions (HOI) is critical for applications ranging from immersive AR/VR to dexterous robotics. Existing methods struggle with generalization, performing well on closed-set objects and predefined tasks but failing to handle unseen objects or open-vocabulary instructions. We introduce OpenHOI, the first framework for open-world HOI synthesis, capable of generating long-horizon manipulation sequences for novel objects guided by free-form language commands. Our approach integrates a 3D Multimodal Large Language Model (MLLM) fine-tuned for joint affordance grounding and semantic task decomposition, enabling precise localization of interaction regions (e.g., handles, buttons) and breakdown of complex instructions (e.g., "Find a water bottle and take a sip") into executable sub-tasks. To synthesize physically plausible interactions, we propose an affordance-driven diffusion model paired with a training-free physics refinement stage that minimizes penetration and optimizes affordance alignment. Evaluations across diverse scenarios demonstrate OpenHOI's superiority over state-of-the-art methods in generalizing to novel object categories, multi-stage tasks, and complex language instructions. Our project page at \href{https://openhoi.github.io}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
