arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.08649cs.CV
收藏

E-Motion:基于事件序列扩散的未来运动模拟

E-Motion: Future Motion Simulation via Event Sequence Diffusion

Xidian University(西安电子科技大学);City University of Hong Kong(香港城市大学)

Song Wu, Zhiyu Zhu, Junhui Hou, Guangming Shi, Jinjian Wu

AI总结:

提出将视频扩散模型与事件相机信息结合的运动模拟框架,利用预训练模型迁移和强化学习对齐机制,实现高精度未来运动预测。

摘要:

预测典型对象的未来运动是计算机视觉中解释和与动态环境交互的关键任务。基于事件的传感器能够以极高的时间粒度捕捉场景变化,可能为预测未来运动提供前所未有的细节和精度机会。受此启发,我们提出将视频扩散模型的强大学习能力与事件相机的丰富运动信息相结合,作为一种运动模拟框架。具体而言,我们首先采用预训练的稳定视频扩散模型来适应事件序列数据集。该过程促进了从RGB视频到以事件为中心的领域的广泛知识迁移。此外,我们引入了一种利用强化学习技术的对齐机制,以增强扩散模型的反向生成轨迹,确保性能和准确性的提升。通过广泛的测试和验证,我们证明了我们的方法在各种复杂场景中的有效性,展示了其在自动驾驶车辆引导、机器人导航和交互媒体等计算机视觉应用中彻底改变运动流预测的潜力。我们的发现为未来研究增强计算机视觉系统的解释能力和预测准确性指明了有前景的方向。

英文摘要:

Forecasting a typical object's future motion is a critical task for interpreting and interacting with dynamic environments in computer vision. Event-based sensors, which could capture changes in the scene with exceptional temporal granularity, may potentially offer a unique opportunity to predict future motion with a level of detail and precision previously unachievable. Inspired by that, we propose to integrate the strong learning capacity of the video diffusion model with the rich motion information of an event camera as a motion simulation framework. Specifically, we initially employ pre-trained stable video diffusion models to adapt the event sequence dataset. This process facilitates the transfer of extensive knowledge from RGB videos to an event-centric domain. Moreover, we introduce an alignment mechanism that utilizes reinforcement learning techniques to enhance the reverse generation trajectory of the diffusion model, ensuring improved performance and accuracy. Through extensive testing and validation, we demonstrate the effectiveness of our method in various complex scenarios, showcasing its potential to revolutionize motion flow prediction in computer vision applications such as autonomous vehicle guidance, robotic navigation, and interactive media. Our findings suggest a promising direction for future research in enhancing the interpretative power and predictive accuracy of computer vision systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑