Tokenizing images into compact visual representations is a key step in learning efficient and high-quality image generative models. We present a simple diffusion tokenizer (DiTo) that learns compact visual representations for image generation models. Our key insight is that a single learning objective, diffusion L2 loss, can be used for training scalable image tokenizers. Since diffusion is already widely used for image generation, our insight greatly simplifies training such tokenizers. In contrast, current state-of-the-art tokenizers rely on an empirically found combination of heuristics and losses, thus requiring a complex training recipe that relies on non-trivially balancing different losses and pretrained supervised models. We show design decisions, along with theoretical grounding, that enable us to scale DiTo for learning competitive image representations. Our results show that DiTo is a simpler, scalable, and self-supervised alternative to the current state-of-the-art image tokenizer which is supervised. DiTo achieves competitive or better quality than state-of-the-art in image reconstruction and downstream image generation tasks.
ViTok-v2: Scaling Native Resolution Auto-Encoders to 5 Billion Parameters
University of Texas, Austin(德克萨斯大学奥斯汀分校);University of Washington(华盛顿大学);Stanford University(斯坦福大学);Spellbrush;Meta Superintelligence Labs(Meta超智能实验室);Georgia Institute of Technology(佐治亚理工学院)
VFM-VAE: Vision Foundation Models Can Be Good Tokenizers for Latent Diffusion Models
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University(人机混合增强智能国家重点实验室,人工智能与机器人研究院,西安交通大学);Microsoft Research Asia(微软亚洲研究院)
What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion
Shanghai Jiao Tong University(上海交通大学);Alibaba Group(阿里巴巴集团);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院,中国科学院);Beihang University(北航);Sun Yat-sen University(中山大学);Nankai University(南开大学);Shanghai AI Laboratory(上海人工智能实验室)
Ophiuchus: Scalable Modeling of Protein Structures through Hierarchical Coarse-graining SO(3)-Equivariant Autoencoders
Center for Bits and Atoms(比特与原子中心);MIT Media Lab, Molecular Machines(麻省理工学院媒体实验室,分子机器);Atomic Architects(原子建筑师);MIT Research Laboratory of Electronics(麻省理工学院电子研究实验室)
BrainG3N: A Dual-Purpose Tokenizer for Controllable 3D Brain MRI Generation
Department of Biomedical Data Science, Stanford University School of Medicine(斯坦福大学医学院生物医学数据科学系);Department of Mathematical Modelling, Statistics & Bioinformatics, Ghent University(根特大学数学建模、统计与生物信息学系);Department of Electrical Engineering, Stanford University(斯坦福大学电气工程系)
School of Computer Science and Engineering, Southeast University, Nanjing, China(1 南京大学计算机科学与工程学院);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(2 教育部新一代人工智能技术及其跨学科应用关键实验室(东南大学))
Resurrect Mask AutoRegressive Modeling for Efficient and Scalable Image Generation
Nanjing University(南京大学);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Innovation Institute(上海创新研究院);The Chinese University of Hong Kong(香港中文大学);Shanghai Jiao Tong University(上海交通大学);University of Minnesota Twin Cities(明尼苏达大学 Twin Cities 分校)
P3P: Pseudo-3D Pre-training for Scaling 3D Voxel-based Masked Autoencoders
SIGS, Tsinghua University(清华大学信号与信息处理系);Youtu Lab, Tencent(腾讯优图实验室);The University of Texas at Austin(德克萨斯大学奥斯汀分校);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Nautilus: Locality-aware Autoencoder for Scalable Mesh Generation
Nanyang Technological University(南洋理工大学);Tencent Hunyuan(腾讯 Hunyuan);South China University of Technology(华南理工大学);The Hong Kong Polytechnic University(香港理工大学);Hong Kong University of Science and Technology(香港科技大学)
Scalable Audio-Visual Masked Autoencoders for Efficient Affective Video Facial Analysis
School of Computer Science and Technology, Xi’an Jiaotong University(西安交通大学计算机科学与技术学院);School of Cyber Science and Engineering, Zhengzhou University(郑州大学网络科学与工程学院);School of Software, Northwestern Polytechnical University(西北工业大学软件学院);Institute of Advanced Technology, University of Science and Technology of China(中国科学技术大学先进技术研究院);Inspur Group(Inspur集团);Research Center for Space Computing System, Zhejiang Lab(浙江实验室空间计算系统研究中心)
针对情感视频面部分析数据有限及跨模态建模难题,提出可扩展音视频 MAE 模型 AVF-MAE++,通过双重掩码、相关性学习模块和三阶段训练,在17个数据集上取得SOTA。