arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2311.17754cs.CVcs.GRcs.HCcs.MM
收藏

基于NeRF的可微分拍摄的电影级行为迁移

Cinematic Behavior Transfer via NeRF-based Differentiable Filming

Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Stanford University(斯坦福大学);The Chinese University of Hong Kong(香港中文大学)

Xuekun Jiang, Anyi Rao, Jingbo Wang, Dahua Lin, Bo Dai

AI总结:

本文提出逆向拍摄行为估计技术,利用NeRF可微分渲染优化摄像机轨迹与SMPL轨迹,并构建电影级迁移流程,将多种镜头类型迁移至2D视频或3D虚拟环境,在用户研究中取得更高评分。

摘要:

在数字媒体和视频制作不断演进的背景下,对摄像机运动和角色动作等视觉元素进行精确操控与复现的需求日益强烈。现有的SLAM方法在动态场景中存在局限性,而人体姿态估计往往只关注2D投影,忽略了3D状态。为解决这些问题,我们首先提出一种逆向拍摄行为估计技术。该技术利用NeRF作为可微分渲染器,并通过优化SMPL轨迹来优化摄像机轨迹。随后,我们提出一种电影级迁移流程,能够将各种镜头类型迁移到新的2D视频或3D虚拟环境中。3D引擎工作流的引入带来了更优越的渲染和控制能力,同时在用户研究中也获得了更高的评分。

英文摘要:

In the evolving landscape of digital media and video production, the precise manipulation and reproduction of visual elements like camera movements and character actions are highly desired. Existing SLAM methods face limitations in dynamic scenes and human pose estimation often focuses on 2D projections, neglecting 3D statuses. To address these issues, we first introduce a reverse filming behavior estimation technique. It optimizes camera trajectories by leveraging NeRF as a differentiable renderer and refining SMPL tracks. We then introduce a cinematic transfer pipeline that is able to transfer various shot types to a new 2D video or a 3D virtual environment. The incorporation of 3D engine workflow enables superior rendering and control abilities, which also achieves a higher rating in the user study.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑