面向长视频端到端生成建模的高效记忆双向Transformer
Towards End-to-End Generative Modeling of Long Videos with Memory-Efficient Bidirectional Transformers
AI总结：
本文提出高效记忆双向Transformer（MeBT），通过将上下文token投影为固定数量潜在token并利用交叉注意力并行解码，以线性复杂度端到端学习视频长期依赖，在长视频生成的质量和速度上均优于自回归方法。
摘要：
自回归Transformer在视频生成中已展现出显著成功。然而，由于自注意力的二次复杂度，这些Transformer无法直接学习视频中的长期依赖关系，并且由于自回归过程，其固有地存在推理速度慢和误差传播的问题。在本文中，我们提出了高效记忆双向Transformer（MeBT），用于端到端学习视频中的长期依赖关系并实现快速推理。基于双向Transformer的最新进展，我们的方法学习从部分观测的图像块中并行解码视频的整个时空体。所提出的Transformer通过将可观测的上下文token投影到固定数量的潜在token中，并通过交叉注意力将它们作为条件来解码被掩码的token，从而在编码和解码中均实现了线性时间复杂度。凭借线性复杂度和双向建模的优势，我们的方法在生成中等长度视频的质量和速度上均显著优于自回归Transformer。视频和代码可在 https://sites.google.com/view/mebt-cvpr2023 获取。
英文摘要：
Autoregressive transformers have shown remarkable success in video generation. However, the transformers are prohibited from directly learning the long-term dependency in videos due to the quadratic complexity of self-attention, and inherently suffering from slow inference time and error propagation due to the autoregressive process. In this paper, we propose Memory-efficient Bidirectional Transformer (MeBT) for end-to-end learning of long-term dependency in videos and fast inference. Based on recent advances in bidirectional transformers, our method learns to decode the entire spatio-temporal volume of a video in parallel from partially observed patches. The proposed transformer achieves a linear time complexity in both encoding and decoding, by projecting observable context tokens into a fixed number of latent tokens and conditioning them to decode the masked tokens through the cross-attention. Empowered by linear complexity and bidirectional modeling, our method demonstrates significant improvement over the autoregressive Transformers for generating moderately long videos in both quality and speed. Videos and code are available at https://sites.google.com/view/mebt-cvpr2023 .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
