arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.04299cs.CV

MotionCanvas：基于可控图像到视频生成的电影镜头设计

MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generation

The Chinese University of Hong Kong（香港中文大学）；Adobe Research（奥多比研究院）；Monash University（莫纳什大学）

Jinbo Xing, Long Mai, Cusuh Ham, Jiahui Huang, Aniruddha Mahapatra, Chi-Wing Fu, Tien-Tsin Wong, Feng Liu

备注：
It is best viewed in Acrobat. Project page: https://motion-canvas25.github.io/
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

MotionCanvas将用户驱动的场景感知运动控制集成到图像到视频生成模型中，无需3D训练数据即可实现3D感知的物体与摄像机运动联合控制，用于电影镜头设计。

摘要：

本文提出了一种方法，使用户能够在图像到视频生成的背景下设计电影视频镜头。镜头设计是电影制作的一个关键方面，涉及对场景中摄像机运动和物体运动的精心规划。然而，在现代图像到视频生成系统中实现直观的镜头设计面临两个主要挑战：第一，有效捕捉用户在运动设计上的意图，其中摄像机运动和场景空间中的物体运动必须被联合指定；第二，表示能够被视频扩散模型有效利用的运动信息，以合成图像动画。为应对这些挑战，我们引入了MotionCanvas，这是一种将用户驱动的控制集成到图像到视频（I2V）生成模型中的方法，允许用户以场景感知的方式同时控制物体运动和摄像机运动。通过结合经典计算机图形学与当代视频生成技术的见解，我们展示了在I2V合成中实现3D感知运动控制的能力，而无需昂贵的3D相关训练数据。MotionCanvas使用户能够直观地描绘场景空间中的运动意图，并将其转化为视频扩散模型的时空运动条件信号。我们在广泛的真实世界图像内容和镜头设计场景上证明了该方法的有效性，突显了其在增强数字内容创作中的创意工作流程以及适应各种图像和视频编辑应用方面的潜力。

英文摘要：

This paper presents a method that allows users to design cinematic video shots in the context of image-to-video generation. Shot design, a critical aspect of filmmaking, involves meticulously planning both camera movements and object motions in a scene. However, enabling intuitive shot design in modern image-to-video generation systems presents two main challenges: first, effectively capturing user intentions on the motion design, where both camera movements and scene-space object motions must be specified jointly; and second, representing motion information that can be effectively utilized by a video diffusion model to synthesize the image animations. To address these challenges, we introduce MotionCanvas, a method that integrates user-driven controls into image-to-video (I2V) generation models, allowing users to control both object and camera motions in a scene-aware manner. By connecting insights from classical computer graphics and contemporary video generation techniques, we demonstrate the ability to achieve 3D-aware motion control in I2V synthesis without requiring costly 3D-related training data. MotionCanvas enables users to intuitively depict scene-space motion intentions, and translates them into spatiotemporal motion-conditioning signals for video diffusion models. We demonstrate the effectiveness of our method on a wide range of real-world image content and shot-design scenarios, highlighting its potential to enhance the creative workflows in digital content creation and adapt to various image and video editing applications.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑