The popular VQ-VAE models reconstruct images through learning a discrete codebook but suffer from a significant issue in the rapid quality degradation of image reconstruction as the compression rate rises. One major reason is that a higher compression rate induces more loss of visual signals on the higher frequency spectrum which reflect the details on pixel space. In this paper, a Frequency Complement Module (FCM) architecture is proposed to capture the missing frequency information for enhancing reconstruction quality. The FCM can be easily incorporated into the VQ-VAE structure, and we refer to the new model as Frequency Augmented VAE (FA-VAE). In addition, a Dynamic Spectrum Loss (DSL) is introduced to guide the FCMs to balance between various frequencies dynamically for optimal reconstruction. FA-VAE is further extended to the text-to-image synthesis task, and a Cross-attention Autoregressive Transformer (CAT) is proposed to obtain more precise semantic attributes in texts. Extensive reconstruction experiments with different compression rates are conducted on several benchmark datasets, and the results demonstrate that the proposed FA-VAE is able to restore more faithfully the details compared to SOTA methods. CAT also shows improved generation quality with better image-text semantic alignment.
MultiMAE for Brain MRIs: Robustness to Missing Inputs Using Multi-Modal Masked Autoencoder
Department of Radiation Oncology, TUM University Hospital, Munich, Germany(辐射肿瘤科,技术大学慕尼黑医院,慕尼黑,德国);Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM)(医疗与医学人工智能教研室,技术大学慕尼黑(TUM));TUM University Hospital, Munich, Germany(技术大学慕尼黑医院,慕尼黑,德国);Chair for AI for Image-Guided Diagnosis and Therapy, Technical University of Munich (TUM)(图像引导诊断与治疗人工智能教研室,技术大学慕尼黑(TUM));Munich Center for Machine Learning (MCML), Munich, Germany(慕尼黑机器学习中心(MCML),慕尼黑,德国);Department of Computing, Imperial College London, London, UK(计算系,伦敦帝国学院,伦敦,英国);Deutsches Konsortium für Translationale Krebsforschung (DKTK), Partner Site Munich, Munich, Germany(德国转化癌症研究联盟(DKTK),慕尼黑合作伙伴站点,慕尼黑,德国);Institute of Radiation Medicine (IRM), Department of Radiation Sciences (DRS), Helmholtz Center Munich, Munich, Germany(辐射医学研究所(IRM),辐射科学部门(DRS),慕尼黑海德堡中心,慕尼黑,德国)
SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders
Institute of Software Chinese Academy of Sciences(中国科学院软件研究所);University of Chinese Academy of Sciences(中国科学院大学);JD Explore Academy(京东探索研究院);Renmin University of China(中国人民大学)
Gaussian Process Prior Variational Autoencoder for Endoscopic Videos
Department of Electromechanics, InViLab, University of Antwerp(安特卫普大学机电工程系InViLab实验室);Department of Computer Science, University of Manchester(曼彻斯特大学计算机科学系);Department of Electrical Engineering, Eindhoven University of Technology(埃因霍温理工大学电气工程系)
Dealing with All-stage Missing Modality: Towards A Universal Model with Robust Reconstruction and Personalization
National University of Singapore(新加坡国立大学);Harvard Medical School(哈佛医学院);Massachusetts General Hospital(麻省总医院);National Neuroscience Institute(国家神经科学研究所);Duke-NUS Medical School(杜克-新加坡国立大学医学院)