面向视频修复的缺陷感知掩码 Transformer
Deficiency-Aware Masked Transformer for Video Inpainting
AI总结：
针对跨帧信息缺失的视频修复问题，提出双模态兼容的 DMT 框架，通过图像先验蒸馏、选择性时空注意力和感受野上下文化器提升性能。
摘要：
近期的视频修复方法通过利用光流等显式引导来传播跨帧像素，取得了显著进展。然而，在某些情况下，掩码视频的跨帧循环信息不可用，从而导致缺陷。在这种情形下，模型不再从其他帧借用像素，而是将重点转向求解逆问题。本文提出一种兼容双模态的修复框架，称为 Deficiency-aware Masked Transformer（DMT，缺陷感知掩码 Transformer），它具有三个关键优势。首先，我们预训练图像修复模型 DMT_img，将其作为蒸馏视频模型 DMT_vid 的先验，从而有助于对缺陷情形进行幻觉生成。其次，自注意力模块选择性地融合时空 token，以加速推理并去除噪声信号。第三，DMT 中集成了一个简单而有效的 Receptive Field Contextualizer（感受野上下文化器），进一步提升性能。在 YouTube-VOS 和 DAVIS 数据集上进行的大量实验表明，DMT_vid 显著优于以往方案。代码和视频演示可在 github.com/yeates/DMT 获取。
英文摘要：
Recent video inpainting methods have made remarkable progress by utilizing explicit guidance, such as optical flow, to propagate cross-frame pixels. However, there are cases where cross-frame recurrence of the masked video is not available, resulting in a deficiency. In such situation, instead of borrowing pixels from other frames, the focus of the model shifts towards addressing the inverse problem. In this paper, we introduce a dual-modality-compatible inpainting framework called Deficiency-aware Masked Transformer (DMT), which offers three key advantages. Firstly, we pretrain a image inpainting model DMT_img serve as a prior for distilling the video model DMT_vid, thereby benefiting the hallucination of deficiency cases. Secondly, the self-attention module selectively incorporates spatiotemporal tokens to accelerate inference and remove noise signals. Thirdly, a simple yet effective Receptive Field Contextualizer is integrated into DMT, further improving performance. Extensive experiments conducted on YouTube-VOS and DAVIS datasets demonstrate that DMT_vid significantly outperforms previous solutions. The code and video demonstrations can be found at github.com/yeates/DMT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
