Articulate-Anything：基于视觉-语言基础模型的铰接物体自动建模
Articulate-Anything: Automatic Modeling of Articulated Objects via a Vision-Language Foundation Model
AI总结：
Articulate-Anything利用视觉-语言模型自动将文本、图像和视频输入转化为可交互的铰接3D物体，通过网格检索和actor-critic迭代优化，在PartNet-Mobility上将成功率从8.7-11.6%提升至75%，并支持仿真到真实的机器人操作策略迁移。
摘要：
交互式3D仿真物体在AR/VR、动画和机器人领域至关重要，推动沉浸式体验和先进自动化。然而，创建这些铰接物体需要大量人力和专业知识，限制了其更广泛的应用。为克服这一挑战，我们提出Articulate-Anything，一个能够从文本、图像和视频等多种输入模态自动铰接多样化复杂物体的系统。Articulate-Anything利用视觉-语言模型（VLMs）生成可编译为可交互数字孪生的代码，用于标准3D仿真器。我们的系统通过网格检索机制利用现有3D资产数据集，并采用actor-critic系统迭代地提出、评估和优化物体铰接方案，自我纠正错误以实现稳健结果。定性评估表明，Articulate-Anything能够通过利用丰富的接地输入来铰接复杂甚至模糊的物体可供性。在标准PartNet-Mobility数据集上的大量定量实验中，Articulate-Anything大幅超越先前工作，将成功率从8.7-11.6%提高到75%，为最先进性能树立了新标杆。我们进一步通过从自然场景视频输入生成3D资产来展示系统的实用性，这些资产随后用于训练机器人策略，以在仿真中执行超越基本抓取和放置的精细操作任务。这些策略随后被迁移到真实机器人系统上。
英文摘要：
Interactive 3D simulated objects are crucial in AR/VR, animations, and robotics, driving immersive experiences and advanced automation. However, creating these articulated objects requires extensive human effort and expertise, limiting their broader applications. To overcome this challenge, we present Articulate-Anything, a system that automates the articulation of diverse, complex objects from many input modalities, including text, images, and videos. Articulate-Anything leverages vision-language models (VLMs) to generate code that can be compiled into an interactable digital twin for use in standard 3D simulators. Our system exploits existing 3D asset datasets via a mesh retrieval mechanism, along with an actor-critic system that iteratively proposes, evaluates, and refines solutions for articulating the objects, self-correcting errors to achieve a robust outcome. Qualitative evaluations demonstrate Articulate-Anything's capability to articulate complex and even ambiguous object affordances by leveraging rich grounded inputs. In extensive quantitative experiments on the standard PartNet-Mobility dataset, Articulate-Anything substantially outperforms prior work, increasing the success rate from 8.7-11.6% to 75% and setting a new bar for state-of-the-art performance. We further showcase the utility of our system by generating 3D assets from in-the-wild video inputs, which are then used to train robotic policies for fine-grained manipulation tasks in simulation that go beyond basic pick and place. These policies are then transferred to a real robotic system.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
