arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.08629cs.CV
收藏

扩展动态人-场景交互建模

Scaling Up Dynamic Human-Scene Interaction Modeling

Institute for AI, Peking University(北京大学人工智能研究院);Beijing Institute of Technology(北京理工大学);National Key Lab of General AI, BIGAI(北京通用人工智能研究院国家重点实验室)

Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, Siyuan Huang

AI总结:

针对人-场景交互建模中的数据稀缺与运动合成难题,提出TRUMANS数据集和基于扩散的自回归模型,实现任意长度HSI序列的高效生成,并在多个3D场景数据集上展现零样本泛化能力。

摘要:

面对人-场景交互建模中数据稀缺和高级运动合成的挑战,我们引入了TRUMANS数据集以及一种新颖的HSI运动合成方法。TRUMANS是目前最全面的动捕HSI数据集,涵盖100个室内场景中超过15小时的人机交互。它精细地捕捉了全身人体运动和物体部件级动态,重点关注接触的真实性。通过将物理环境转换为精确的虚拟模型,并对人和物体的外观和运动进行广泛增强,同时保持交互保真度,该数据集得以进一步扩展。利用TRUMANS,我们设计了一种基于扩散的自回归模型,能够高效生成任意长度的HSI序列,同时考虑场景上下文和预期动作。在实验中,我们的方法在一系列3D场景数据集(如PROX、Replica、ScanNet、ScanNet++)上表现出显著的零样本泛化能力,生成的运动与原始动捕序列高度相似,定量实验和人类研究均证实了这一点。

英文摘要:

Confronting the challenges of data scarcity and advanced motion synthesis in human-scene interaction modeling, we introduce the TRUMANS dataset alongside a novel HSI motion synthesis method. TRUMANS stands as the most comprehensive motion-captured HSI dataset currently available, encompassing over 15 hours of human interactions across 100 indoor scenes. It intricately captures whole-body human motions and part-level object dynamics, focusing on the realism of contact. This dataset is further scaled up by transforming physical environments into exact virtual models and applying extensive augmentations to appearance and motion for both humans and objects while maintaining interaction fidelity. Utilizing TRUMANS, we devise a diffusion-based autoregressive model that efficiently generates HSI sequences of any length, taking into account both scene context and intended actions. In experiments, our approach shows remarkable zero-shot generalizability on a range of 3D scene datasets (e.g., PROX, Replica, ScanNet, ScanNet++), producing motions that closely mimic original motion-captured sequences, as confirmed by quantitative experiments and human studies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑