AGILE: 通过代理生成从视频重建手-物体交互
AGILE: Hand-Object Interaction Reconstruction from Video via Agentic Generation
AI总结：
提出AGILE框架，利用视觉语言模型引导生成完整物体网格，结合锚定-跟踪策略和接触感知优化，从单目视频鲁棒重建手-物体交互，生成可直接用于仿真的资产。
摘要：
从单目视频重建动态手-物体交互对于灵巧操作数据收集以及为机器人和VR创建逼真的数字孪生至关重要。然而，当前方法面临两个难以逾越的障碍：(1) 依赖神经渲染通常在严重遮挡下产生碎片化、不可用于仿真的几何体；(2) 依赖脆弱的运动恢复结构（SfM）初始化导致在野外视频中频繁失败。为克服这些限制，我们提出AGILE，一个鲁棒的框架，将范式从重建转变为交互学习的代理生成。首先，我们采用代理流水线，其中视觉语言模型（VLM）引导生成模型合成一个完整、水密的物体网格，具有高保真纹理，不受视频遮挡影响。其次，完全绕过脆弱的SfM，我们提出一种鲁棒的锚定-跟踪策略。我们使用基础模型在单个交互起始帧初始化物体姿态，并通过利用生成资产与视频观测之间的强视觉相似性在时间上传播姿态。最后，接触感知优化整合语义、几何和交互稳定性约束以强制执行物理合理性。在HO3D、DexYCB、ARCTIC和野外视频上的大量实验表明，AGILE在全局几何精度上优于基线，同时在先前技术经常崩溃的具有挑战性的序列上表现出卓越的鲁棒性。通过优先考虑物理有效性，我们的方法生成可直接用于仿真的资产，并通过真实到仿真重定向在机器人应用中验证。项目页面：https://agile-hoi.github.io。
英文摘要：
Reconstructing dynamic hand-object interactions from monocular videos is critical for dexterous manipulation data collection and creating realistic digital twins for robotics and VR. However, current methods face two prohibitive barriers: (1) reliance on neural rendering often yields fragmented, non-simulation-ready geometries under heavy occlusion, and (2) dependence on brittle Structure-from-Motion (SfM) initialization leads to frequent failures on in-the-wild footage. To overcome these limitations, we introduce AGILE, a robust framework that shifts the paradigm from reconstruction to agentic generation for interaction learning. First, we employ an agentic pipeline where a Vision-Language Model (VLM) guides a generative model to synthesize a complete, watertight object mesh with high-fidelity texture, independent of video occlusions. Second, bypassing fragile SfM entirely, we propose a robust anchor-and-track strategy. We initialize the object pose at a single interaction onset frame using a foundation model and propagate it temporally by leveraging the strong visual similarity between our generated asset and video observations. Finally, a contact-aware optimization integrates semantic, geometric, and interaction stability constraints to enforce physical plausibility. Extensive experiments on HO3D, DexYCB, ARCTIC, and in-the-wild videos reveal that AGILE outperforms baselines in global geometric accuracy while demonstrating exceptional robustness on challenging sequences where prior arts frequently collapse. By prioritizing physical validity, our method produces simulation-ready assets validated via real-to-sim retargeting for robotic applications. Project page: https://agile-hoi.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
