arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.09494cs.CVcs.LG
收藏

Transframer:使用生成模型的任意帧预测

Transframer: Arbitrary Frame Prediction with Generative Models

DeepMind(深度思维)

Charlie Nash, João Carreira, Jacob Walker, Iain Barr, Andrew Jaegle, Mateusz Malinowski, Peter Battaglia

摘要:

我们提出了一个通用的框架,图像建模和视觉任务的基础上概率帧预测。我们的方法统一了广泛的任务,从图像分割,到新颖的视图合成和视频插值。我们将这个框架与我们称为Transframer的架构配对,该架构使用U-Net和Transformer组件来调节注释的上下文框架,并输出稀疏的压缩图像特征序列。Transframer是各种视频生成基准的最新技术,在Few-Shot视图合成方面与最强的模型竞争,并且可以从单个图像生成连贯的30秒视频,而无需任何明确的几何信息。一个通用的Transframer同时产生8个任务,包括语义分割,图像分类和光流预测,没有特定任务的架构组件,表明多任务计算机视觉可以使用概率图像模型来解决。我们的方法原则上可以应用于广泛的应用程序,需要学习的条件结构的注释图像格式的数据。

英文摘要:

We present a general-purpose framework for image modelling and vision tasks based on probabilistic frame prediction. Our approach unifies a broad range of tasks, from image segmentation, to novel view synthesis and video interpolation. We pair this framework with an architecture we term Transframer, which uses U-Net and Transformer components to condition on annotated context frames, and outputs sequences of sparse, compressed image features. Transframer is the state-of-the-art on a variety of video generation benchmarks, is competitive with the strongest models on few-shot view synthesis, and can generate coherent 30 second videos from a single image without any explicit geometric information. A single generalist Transframer simultaneously produces promising results on 8 tasks, including semantic segmentation, image classification and optical flow prediction with no task-specific architectural components, demonstrating that multi-task computer vision can be tackled using probabilistic image models. Our approach can in principle be applied to a wide range of applications that require learning the conditional structure of annotated image-formatted data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑