arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.03069cs.CVcs.AI
收藏

TokenFlow:用于多模态理解与生成的统一图像分词器

TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation

ByteDance(字节跳动)

Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K. Du, Zehuan Yuan, Xinglong Wu

AI总结:

TokenFlow提出双码本统一图像分词器,解耦语义与像素特征,首次使离散视觉输入在理解上超越LLaVA-1.5 13B(平均提升7.2%),并在重建(FID 0.63)和生成(GenEval 0.55)上达到领先水平。

摘要:

我们提出了TokenFlow,一种新颖的统一图像分词器,它弥合了多模态理解与生成之间长期存在的鸿沟。先前的研究尝试采用单一的面向重建的向量量化(VQ)编码器来统一这两项任务。我们观察到,理解与生成需要根本不同粒度的视觉信息。这导致了关键的权衡,尤其在多模态理解任务中损害了性能。TokenFlow通过一种创新的双码本架构解决了这一挑战,该架构解耦了语义级和像素级特征学习,同时通过共享映射机制保持它们的对齐。这种设计使得通过共享索引能够直接访问对理解任务至关重要的高层语义表示和对生成所必需的细粒度视觉特征。我们的大量实验证明了TokenFlow在多个维度上的优越性。借助TokenFlow,我们首次证明离散视觉输入在理解性能上可以超越LLaVA-1.5 13B,实现了7.2%的平均提升。对于图像重建,我们在384*384分辨率下取得了0.63的强FID分数。此外,TokenFlow在自回归图像生成中确立了最先进的性能,在256*256分辨率下GenEval分数达到0.55,与SDXL取得了相当的结果。

英文摘要:

We present TokenFlow, a novel unified image tokenizer that bridges the long-standing gap between multimodal understanding and generation. Prior research attempt to employ a single reconstruction-targeted Vector Quantization (VQ) encoder for unifying these two tasks. We observe that understanding and generation require fundamentally different granularities of visual information. This leads to a critical trade-off, particularly compromising performance in multimodal understanding tasks. TokenFlow addresses this challenge through an innovative dual-codebook architecture that decouples semantic and pixel-level feature learning while maintaining their alignment via a shared mapping mechanism. This design enables direct access to both high-level semantic representations crucial for understanding tasks and fine-grained visual features essential for generation through shared indices. Our extensive experiments demonstrate TokenFlow's superiority across multiple dimensions. Leveraging TokenFlow, we demonstrate for the first time that discrete visual input can surpass LLaVA-1.5 13B in understanding performance, achieving a 7.2\% average improvement. For image reconstruction, we achieve a strong FID score of 0.63 at 384*384 resolution. Moreover, TokenFlow establishes state-of-the-art performance in autoregressive image generation with a GenEval score of 0.55 at 256*256 resolution, achieving comparable results to SDXL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑