arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.14178cs.CVcs.AI
收藏

UniWeTok: 一种具有 $2^{128}$ 大小代码本的统一二进制分词器用于统一多模态大语言模型

UniWeTok: An Unified Binary Tokenizer with Codebook Size $\mathit{2^{128}}$ for Unified Multimodal Large Language Model

Shaobin Zhuang, Yuang Ai, Jiaming Han, Weijia Mao, Xiaohui Li, Fangyikang Wang, Xiao Wang, Yan Li, Shanchuan Lin, Kun Xu, Zhenheng Yang, Huaibo Huang, Xiangyu Y… 展开作者

Shaobin Zhuang, Yuang Ai, Jiaming Han, Weijia Mao, Xiaohui Li, Fangyikang Wang, Xiao Wang, Yan Li, Shanchuan Lin, Kun Xu, Zhenheng Yang, Huaibo Huang, Xiangyu Yue, Hao Chen, Yali Wang

AI总结:

UniWeTok通过大规模二进制代码本和改进的训练框架,实现统一多模态大语言模型的高效视觉表示。

摘要:

统一多模态大语言模型(MLLMs)需要一种视觉表示,能够同时支持高保真重建、复杂语义提取和生成适宜性。然而,现有的视觉分词器通常难以在单一框架内满足这些相互冲突的目标。在本文中,我们介绍了UniWeTok,一种统一的离散分词器,通过使用大规模二进制代码本($2^{128}$)来弥合这一差距。在训练框架方面,我们引入了预-后蒸馏和生成意识先验,以增强离散标记的语义提取和生成先验。在模型架构方面,我们提出了卷积-注意力混合架构,并引入了SigLu激活函数。SigLu激活不仅限制了编码器输出并稳定了语义蒸馏过程,还有效解决了标记熵损失与承诺损失之间的优化冲突。我们进一步提出了一个三阶段训练框架,旨在增强UniWeTok在各种图像分辨率和感知敏感场景中的适应性,如涉及人类面孔和文本内容的场景。在ImageNet上,UniWeTok在图像生成性能(FID:UniWeTok 1.38 vs. REPA 1.42)方面达到最先进的水平,同时训练计算量显著较低(训练标记:UniWeTok 33B vs. REPA 262B)。在通用领域,UniWeTok在广泛的任务中表现出高度竞争的能力,包括多模态理解、图像生成(DPG Score:UniWeTok 86.63 vs. FLUX.1 [Dev] 83.84)和编辑(GEdit Overall Score:UniWeTok 5.09 vs. OmniGen 5.06)。我们发布了代码和模型,以促进社区对统一分词器和MLLM的探索。

英文摘要:

Unified Multimodal Large Language Models (MLLMs) require a visual representation that simultaneously supports high-fidelity reconstruction, complex semantic extraction, and generative suitability. However, existing visual tokenizers typically struggle to satisfy these conflicting objectives within a single framework. In this paper, we introduce UniWeTok, a unified discrete tokenizer designed to bridge this gap using a massive binary codebook ($\mathit{2^{128}}$). For training framework, we introduce Pre-Post Distillation and a Generative-Aware Prior to enhance the semantic extraction and generative prior of the discrete tokens. In terms of model architecture, we propose a convolution-attention hybrid architecture with the SigLu activation function. SigLu activation not only bounds the encoder output and stabilizes the semantic distillation process but also effectively addresses the optimization conflict between token entropy loss and commitment loss. We further propose a three-stage training framework designed to enhance UniWeTok's adaptability cross various image resolutions and perception-sensitive scenarios, such as those involving human faces and textual content. On ImageNet, UniWeTok achieves state-of-the-art image generation performance (FID: UniWeTok 1.38 vs. REPA 1.42) while requiring a remarkably low training compute (Training Tokens: UniWeTok 33B vs. REPA 262B). On general-domain, UniWeTok demonstrates highly competitive capabilities across a broad range of tasks, including multimodal understanding, image generation (DPG Score: UniWeTok 86.63 vs. FLUX.1 [Dev] 83.84), and editing (GEdit Overall Score: UniWeTok 5.09 vs. OmniGen 5.06). We release code and models to facilitate community exploration of unified tokenizer and MLLM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑