FiT: Flexible Vision Transformer for Diffusion Model
Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Tsinghua University(清华大学);Sydney University(悉尼大学);The University of Hong Kong(香港大学)
Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, Lei Bai
Nature is infinitely resolution-free. In the context of this reality, existing diffusion models, such as Diffusion Transformers, often face challenges when processing image resolutions outside of their trained domain. To overcome this limitation, we present the Flexible Vision Transformer (FiT), a transformer architecture specifically designed for generating images with unrestricted resolutions and aspect ratios. Unlike traditional methods that perceive images as static-resolution grids, FiT conceptualizes images as sequences of dynamically-sized tokens. This perspective enables a flexible training strategy that effortlessly adapts to diverse aspect ratios during both training and inference phases, thus promoting resolution generalization and eliminating biases induced by image cropping. Enhanced by a meticulously adjusted network structure and the integration of training-free extrapolation techniques, FiT exhibits remarkable flexibility in resolution extrapolation generation. Comprehensive experiments demonstrate the exceptional performance of FiT across a broad range of resolutions, showcasing its effectiveness both within and beyond its training resolution distribution. Repository available at https://github.com/whlzy/FiT.
GeoRelight: Learning Joint Geometrical Relighting and Reconstruction with Flexible Multi-Modal Diffusion Transformers
Codec Avatars Lab, Meta(Meta编码器动画实验室);University of Tübingen(图宾根大学);Max Planck Institute for Informatics, Saarland Informatics Campus(马克斯·普朗克信息学院,萨尔兰信息校园)
VolDiT: Controllable Volumetric Medical Image Synthesis with Diffusion Transformers
Institute for Artificial Intelligence in Cardiovascular Medicine, Medical Faculty of Heidelberg University, Heidelberg University(海德堡大学医学院心血管医学人工智能研究所);Department of Cardiology, Angiology, Pneumology, Heidelberg University Hospital(海德堡大学医院心脏病学、血管学、肺病学系);DZHK (German Centre for Cardiovascular Research), Partner Site Heidelberg/Mannheim(德国心血管研究中心(DZHK)海德堡/曼海姆合作站点);University Medical Center of the Johannes Gutenberg-University Mainz(美因茨约翰内斯·古腾堡大学医学中心)
FM-ViT: Flexible Modal Vision Transformers for Face Anti-Spoofing
Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所);Baidu Research(百度研究院);Great Bay University(大湾区大学);Mininglamp Academy of Sciences, Mininglamp Technology(明略科技明略科学院);Macau University of Science and Technology(澳门科技大学);Westlake University(西湖大学)
There and Back Again: A Flexible-Frame Transformer for Multi-Exposure Fusion
Nankai International Advanced Research Institute (SHENZHEN·FUTIAN)(南开国际先进研究院(深圳·福田));Peng Cheng Laboratory(鹏城实验室);College of Computer Science, Nankai University(南开大学计算机学院);The Hong Kong Polytechnic University(香港理工大学);OPPO Research Institute(OPPO研究院)
Surface Vision Transformers: Flexible Attention-Based Modelling of Biomedical Surfaces
King’s College London(伦敦国王学院);Washington University Medical School(华盛顿大学医学院);Imperial College London(帝国理工学院);Technical University of Munich(慕尼黑工业大学);University of Edinburgh(爱丁堡大学)
Beihang University(北京航空航天大学);Johns Hopkins University(约翰斯·霍普金斯大学);Hangzhou Innovation Institute of Beihang University(北京航空航天大学杭州创新研究院);Zhengzhou University(郑州大学);University of Science and Technology of China(中国科学技术大学);Xidian University(西安电子科技大学)