We present TokenFlow, a novel unified image tokenizer that bridges the long-standing gap between multimodal understanding and generation. Prior research attempt to employ a single reconstruction-targeted Vector Quantization (VQ) encoder for unifying these two tasks. We observe that understanding and generation require fundamentally different granularities of visual information. This leads to a critical trade-off, particularly compromising performance in multimodal understanding tasks. TokenFlow addresses this challenge through an innovative dual-codebook architecture that decouples semantic and pixel-level feature learning while maintaining their alignment via a shared mapping mechanism. This design enables direct access to both high-level semantic representations crucial for understanding tasks and fine-grained visual features essential for generation through shared indices. Our extensive experiments demonstrate TokenFlow's superiority across multiple dimensions. Leveraging TokenFlow, we demonstrate for the first time that discrete visual input can surpass LLaVA-1.5 13B in understanding performance, achieving a 7.2\% average improvement. For image reconstruction, we achieve a strong FID score of 0.63 at 384*384 resolution. Moreover, TokenFlow establishes state-of-the-art performance in autoregressive image generation with a GenEval score of 0.55 at 256*256 resolution, achieving comparable results to SDXL.
ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representations
Shanghai Key Lab of Intelligent Information Processing, Fudan University(复旦大学上海智能信息处理重点实验室);School of Computer Science, Fudan University(复旦大学计算机科学技术学院);Shanghai Collaborative Innovation Center of Intelligent Visual Computing(上海智能视觉计算协同创新中心);Youtu Lab, Tencent(腾讯优图实验室);Meta AI;Shanghai AI Laboratory(上海人工智能实验室)
UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Beihang University(北京航空航天大学);Shenzhen Key Lab of Computer Vision and Pattern Recognition, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(深圳计算机视觉与模式识别重点实验室,深圳先进技术研究院,中国科学院);Nanjing University(南京大学);University of Science and Technology of China(中国科学技术大学);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces
Machine Learning Center, Georgia Institute of Technology, Atlanta, GA;School of Mathematics, Georgia Institute of Technology, Atlanta, GA;Department of Mathematics \& Statistics, SUNY Albany, NY
AnyMo: Scaling Any-Modality Conditional Motion Generation with Masked Modeling
Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, China(中国科学院智能信息处理重点实验室(中国科学院计算技术研究所,中国));University of Chinese Academy of Sciences, China(中国科学院大学)
OmniGen2: Towards Instruction-Aligned Multimodal Generation
Beijing Academy of Artificial Intelligence(北京人工智能研究院);University of Science and Technology of China(中国科学技术大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Zhejiang University(浙江大学)
M$^3$GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation
Peng Cheng Laboratory(鹏城实验室);University of Chinese Academy of Sciences(中国科学院大学);WeChat, Tencent Inc(腾讯公司微信事业部);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学深圳校区)
LottieGPT: Tokenizing Vector Animation for Autoregressive Generation
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);AIR, Tsinghua University(清华大学人工智能研究院);BAAI(百度人工智能研究院);The Hong Kong Polytechnic University(香港理工大学);Nanjing University(南京大学);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济实验室(深圳))
TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation
ARC Lab, Tencent PCG(腾讯PCG广告实验室);City University of Hong Kong(香港城市大学);Zhejiang University(浙江大学);NLPR & MAIS, Institute of Automation, CAS, Beijing(自动化研究所)
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Dalian University of Technology(大连理工大学);Tsinghua University(清华大学)
针对视觉分词器与下游任务优化脱节导致的表示瓶颈,提出 ETT 端到端联合调优方法,在多模态理解与生成任务上较冻结基线提升 2-6%。
Seeing the Forest and the Trees: Query-Aware Tokenizer for Long-Video Multimodal Language Models
Queen Mary University of London(伦敦女王学院);University of Sheffield(谢菲尔德大学);Imperial College London(伦敦帝国学院);Pengcheng Laboratory(鹏城实验室);Meta Inc(Meta公司);Meituan Inc(美团公司);Nanjing University of Science(南京理工大学);University of Birmingham(伯明翰大学);Utrecht University(乌得勒支大学)
From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
School of Computer Science, Peking University(北京大学计算机学院);The Chinese University of Hong Kong(香港中文大学);University of California, San Diego(加利福尼亚大学圣迭戈分校);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Image Tokens Matter: Mitigating Hallucination in Discrete Tokenizer-based Large Vision-Language Models via Latent Editing
Hasso Plattner Institute(霍普夫纳研究所);University of Potsdam(波茨坦大学);University of Cambridge(剑桥大学);University of Oxford(牛津大学);German University of Digital Science(德国数字科学大学)