arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2211.02145cs.CV
收藏

FactorMatte:重新定义视频遮片,用于重新合成任务

FactorMatte: Redefining Video Matting for Re-Composition Tasks

Cornell Tech(康奈尔科技校区);Cornell University(康奈尔大学)

Zeqi Gu, Wenqi Xian, Noah Snavely, Abe Davis

摘要:

我们提出了“因子抠像”,一种替代配方的视频抠像问题的反事实的视频合成,更适合于重组任务。因子抠图的目标是将视频内容分离成独立的组件,每个组件都可视化场景的反事实版本,其中其他组件的内容已被删除。我们表明,因子抠图映射到一个更一般的贝叶斯框架的抠图问题,占复杂的条件层之间的相互作用。基于这一观察,我们提出了一种解决因子抠图问题的方法,该方法即使对于具有复杂跨层交互(如飞溅、阴影和反射)的视频也能产生有用的分解。我们的方法是按视频训练的,既不需要外部大型数据集的预训练,也不需要场景的3D结构的知识。我们进行了大量的实验,并表明我们的方法不仅可以解开复杂的交互场景,而且在现有的任务,如经典的视频抠图和背景减除的顶级方法。此外,我们还展示了我们的方法在一系列下游任务上的优势。请参阅我们的项目网页了解更多详情:https://factormatte.github.io

英文摘要:

We propose "factor matting", an alternative formulation of the video matting problem in terms of counterfactual video synthesis that is better suited for re-composition tasks. The goal of factor matting is to separate the contents of video into independent components, each visualizing a counterfactual version of the scene where contents of other components have been removed. We show that factor matting maps well to a more general Bayesian framing of the matting problem that accounts for complex conditional interactions between layers. Based on this observation, we present a method for solving the factor matting problem that produces useful decompositions even for video with complex cross-layer interactions like splashes, shadows, and reflections. Our method is trained per-video and requires neither pre-training on external large datasets, nor knowledge about the 3D structure of the scene. We conduct extensive experiments, and show that our method not only can disentangle scenes with complex interactions, but also outperforms top methods on existing tasks such as classical video matting and background subtraction. In addition, we demonstrate the benefits of our approach on a range of downstream tasks. Please refer to our project webpage for more details: https://factormatte.github.io

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑