arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.16804cs.CV
收藏

InTraGen:基于轨迹的视频生成用于物体交互

InTraGen: Trajectory-controlled Video Generation for Object Interactions

ETH Zurich(苏黎世联邦理工学院);Sun Yat-sen University(中山大学);Aalborg University(奥尔堡大学)

Zuhao Liu, Aleksandar Yanev, Ahmad Mahmood, Ivan Nikolov, Saman Motamed, Wei-Shi Zheng, Xi Wang, Lei Sun, Luc Van Gool, Danda Pani Paudel

AI总结:

本文提出InTraGen,通过引入四个新数据集和轨迹质量度量指标,改进多物体交互场景的轨迹引导视频生成,提升视觉真实性和定量性能。

摘要:

视频生成的进步显著提升了生成场景的真实性和质量。这激发了开发直观工具的兴趣,使用户能够利用视频生成作为世界模拟器。文本到视频(T2V)生成是一种此类方法,仅通过文本描述生成视频。然而,由于文本本身的模糊性和文本提示提供的有限时间信息,研究人员探索了额外的控制信号,如轨迹引导系统,以实现更准确的T2V生成。然而,评估T2V模型能否生成多个物体之间真实交互的方法仍然不足。我们介绍了InTraGen,一个改进轨迹引导生成物体交互场景的管道。我们提出了4个新数据集和一个新颖的轨迹质量度量指标,以评估所提出的InTraGen的性能。为了实现物体交互,我们引入了多模态交互编码管道,包含一个物体ID注入机制,以丰富物体与环境的交互。我们的结果表明在视觉真实性和定量性能方面都有所改进。代码和数据集可在https://github.com/insait-institute/InTraGen获取。

英文摘要:

Advances in video generation have significantly improved the realism and quality of created scenes. This has fueled interest in developing intuitive tools that let users leverage video generation as world simulators. Text-to-video (T2V) generation is one such approach, enabling video creation from text descriptions only. Yet, due to the inherent ambiguity in texts and the limited temporal information offered by text prompts, researchers have explored additional control signals like trajectory-guided systems, for more accurate T2V generation. Nonetheless, methods to evaluate whether T2V models can generate realistic interactions between multiple objects are lacking. We introduce InTraGen, a pipeline for improved trajectory-based generation of object interaction scenarios. We propose 4 new datasets and a novel trajectory quality metric to evaluate the performance of the proposed InTraGen. To achieve object interaction, we introduce a multi-modal interaction encoding pipeline with an object ID injection mechanism that enriches object-environment interactions. Our results demonstrate improvements in both visual fidelity and quantitative performance. Code and datasets are available at https://github.com/insait-institute/InTraGen

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑