Video outpainting aims to adequately complete missing areas at the edges of video frames. Compared to image outpainting, it presents an additional challenge as the model should maintain the temporal consistency of the filled area. In this paper, we introduce a masked 3D diffusion model for video outpainting. We use the technique of mask modeling to train the 3D diffusion model. This allows us to use multiple guide frames to connect the results of multiple video clip inferences, thus ensuring temporal consistency and reducing jitter between adjacent frames. Meanwhile, we extract the global frames of the video as prompts and guide the model to obtain information other than the current video clip using cross-attention. We also introduce a hybrid coarse-to-fine inference pipeline to alleviate the artifact accumulation problem. The existing coarse-to-fine pipeline only uses the infilling strategy, which brings degradation because the time interval of the sparse frames is too large. Our pipeline benefits from bidirectional learning of the mask modeling and thus can employ a hybrid strategy of infilling and interpolation when generating sparse frames. Experiments show that our method achieves state-of-the-art results in video outpainting tasks. More results and codes are provided at our https://fanfanda.github.io/M3DDM/.
OutDreamer: Video Outpainting with a Diffusion Transformer
Shanghai Jiao Tong University(上海交通大学);Huawei Noah’s Ark Lab(华为诺亚实验室);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院,中国科学院)
Conditional Panoramic Image Generation via Masked Autoregressive Modeling
School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);Insta360 Research(Insta360研究院);National University of Singapore(新加坡国立大学);The University of Tokyo(东京大学)
GlobalPaint: Spatiotemporal Coherent Video Outpainting with Global Feature Guidance
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University(人机混合增强智能国家重点实验室,人工智能与机器人研究院,西安交通大学);Microsoft Research Asia(微软亚洲研究院);University of Science and Technology of China(中国科学技术大学)
BiGR: Harnessing Binary Latent Codes for Image Generation and Improved Visual Representation Capabilities
The University of Hong Kong(香港大学);Hong Kong University of Science and Technology(香港科技大学);Intellifusion(云天励飞);The Chinese University of Hong Kong(香港中文大学)
HMSViT: A Hierarchical Masked Self-Supervised Vision Transformer for Corneal Nerve Segmentation and Diabetic Neuropathy Diagnosis
Department of Computing and Mathematics, Manchester Metropolitan University(计算与数学系,曼彻斯特 Metropolitan 大学);Department of Eye and Vision Sciences, University of Liverpool(眼科学与视觉科学系,利物浦大学);Faculty of Biology, Medicine and Health, University of Manchester(生物、医学与健康学院,曼彻斯特大学);Department of Medicine, Weill Cornell Medicine-Qatar(医学系,Weill Cornell Medicine-Qatar)
Hi-End-MAE: Hierarchical encoder-driven masked autoencoders are stronger vision learners for medical image segmentation
University of Science and Technology of China (USTC)(中国科学技术大学);Suzhou Institute for Advanced Research, USTC(中国科学技术大学苏州高等研究院);Stanford University(斯坦福大学);State Key Laboratory of Precision and Intelligent Chemistry, USTC(中国科学技术大学精准与智能化学国家重点实验室);Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS(中国科学院计算技术研究所中国科学院智能信息处理重点实验室)
Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training
The Chinese University of Hong Kong(香港中文大学);Shanghai AI Laboratory(上海人工智能实验室);Centre for Perceptual and Interactive Intelligence Limited(感知与交互智能中心有限公司)
HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Department of Automation, Tsinghua University(清华大学自动化系);Beijing National Research Center for Information Science and Technology, Tsinghua University(清华大学北京信息科学与技术国家研究中心)
HERO: Hierarchical Embedding-Refinement for Open-Vocabulary Temporal Sentence Grounding in Videos
Laboratory of Complex Systems Modeling and Simulation, School of Computer Science and Technology, Hangzhou Dianzi University(电子科技大学复杂系统建模与仿真实验室,计算机科学与技术学院);Zhejiang Key Laboratory of Space Information Sensing and Transmission, Hangzhou Dianzi University(浙江省空间信息感知与传输重点实验室,电子科技大学);Department of Psychological and Cognitive Sciences, Tsinghua University(清华大学心理与认知科学系)
Focus-to-Perceive Representation Learning: A Cognition-Inspired Hierarchical Framework for Endoscopic Video Analysis
Key Laboratory of Intelligent Computing and Information Processing of Ministry of Education, Xiangtan University(湘潭大学教育部智能计算与信息处理重点实验室);School of Computer Science, Xiangtan University(湘潭大学计算机科学学院);Key Laboratory for Artificial Intelligence and International Communication, Hunan Normal University(湖南师范大学人工智能与国际传播重点实验室)
EndoMamba: An Efficient Foundation Model for Endoscopic Videos via Hierarchical Pre-training
State Key Laboratory of Multimodal Artificial Intelligence Systems(多模态人工智能系统国家重点实验室);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);The First Affiliated Hospital, Sun Yat-sen University(中山大学第一附属医院);Centre for Artificial Intelligence and Robotics, Chinese Academy of Sciences(中国科学院人工智能与机器人中心);School of Engineering and Imaging Sciences, King’s College London(伦敦大学国王学院工程与影像科学学院)