arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.10090cs.CV
收藏

InteractPro:一种运动感知图像合成的统一框架

InteractPro: A Unified Framework for Motion-Aware Image Composition

College of Computing and Data Science, Nanyang Technological University(计算与数据科学学院,南洋理工大学);DAMO Academy, Alibaba Group(阿里达摩院)

Weijing Tao, Xiaofeng Yang, Miaomiao Cui, Guosheng Lin

AI总结:

InteractPro提出一个统一框架,通过LVLM规划器结合物理模拟与视频扩散,实现可控且连贯的运动感知图像合成,克服传统方法需手动规划且输出静态的局限。

摘要:

我们提出了InteractPro,一个用于动态运动感知图像合成的综合框架。其核心是InteractPlan,一个智能规划器,利用大型视觉语言模型(LVLM)进行场景分析和物体放置,确定实现逼真运动效果的最佳合成策略。基于每个场景,InteractPlan在我们的两个专用模块之间进行选择:InteractPhys和InteractMotion。InteractPhys采用基于增强型物质点法(MPM)的模拟,以产生物理上真实且可控的物体-场景交互,捕捉需要真实物理建模的多样化和抽象事件。相比之下,InteractMotion是一种基于预训练视频扩散的免训练方法。传统的合成方法存在两个主要局限性:需要手动规划物体放置以及生成静态、无运动的输出。通过将基于模拟和基于扩散的方法统一在规划器指导下,InteractPro克服了这些挑战,确保生成具有丰富运动感知的合成结果。大量的定量和定性评估表明,InteractPro在生成跨不同场景的可控且连贯的合成方面具有有效性。

英文摘要:

We introduce InteractPro, a comprehensive framework for dynamic motion-aware image composition. At its core is InteractPlan, an intelligent planner that leverages a Large Vision Language Model (LVLM) for scenario analysis and object placement, determining the optimal composition strategy to achieve realistic motion effects. Based on each scenario, InteractPlan selects between our two specialized modules: InteractPhys and InteractMotion. InteractPhys employs an enhanced Material Point Method (MPM)-based simulation to produce physically faithful and controllable object-scene interactions, capturing diverse and abstract events that require true physical modeling. InteractMotion, in contrast, is a training-free method based on pretrained video diffusion. Traditional composition approaches suffer from two major limitations: requiring manual planning for object placement and generating static, motionless outputs. By unifying simulation-based and diffusion-based methods under planner guidance, InteractPro overcomes these challenges, ensuring richly motion-aware compositions. Extensive quantitative and qualitative evaluations demonstrate InteractPro's effectiveness in producing controllable, and coherent compositions across varied scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑