arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01554cs.LGcs.AIcs.CV
收藏

InfoTok: 基于信息理论的容量受限共享视觉标记化在统一多模态大语言模型中的应用

InfoTok: Information-Theoretic Regularization for Capacity-Constrained Shared Visual Tokenization in Unified MLLMs

University of Alberta(阿尔伯塔大学);vivo Mobile Communication Co., Ltd(维沃移动通信有限公司)

Lv Tang, Tianyi Zheng, Bo Li, Xingyu Li

AI总结:

本文提出InfoTok,一种基于信息瓶颈原理的信息正则化标记化机制,通过互信息约束平衡压缩与任务相关性,提升统一多模态大语言模型的图像理解和生成性能。

摘要:

统一多模态大语言模型(MLLMs)旨在在一个框架内统一图像理解和生成,其中共享视觉标记器作为唯一接口,将高维图像映射到有限的标记预算中,以支持下游多模态推理和生成。然而,现有共享标记设计大多受架构驱动,缺乏明确标准来决定应保留哪些信息以同时支持语义抽象和视觉细节。本文采用容量受限视角,将共享标记器视为计算受限的学习者,其有限的表示预算应优先考虑可重用的结构而非难以利用的高熵变化和冗余。受此观点启发,我们提出InfoTok,一种基于信息瓶颈(IB)原理的信息正则化标记化机制。InfoTok通过施加互信息(MI)约束,控制从图像到共享标记到多模态输出的信息流,从而在压缩与任务相关性之间实现有原则的权衡,同时鼓励跨模态一致性。由于MI对高维视觉表示不可行,我们用实际可微的依赖估计器实例化InfoTok,包括变分IB公式和基于希尔伯特-施密特独立准则(HSIC)的替代方案。将InfoTok集成到三个代表性的统一MLLMs中,无需引入额外训练数据,一致提升了图像理解和生成性能。这些结果支持信息正则化的视觉标记化作为统一MLLMs中标记学习的坚实基础。

英文摘要:

Unified multimodal large language models (MLLMs) aim to unify image understanding and image generation within a single framework, where a shared visual tokenizer serves as the sole interface that maps high-dimensional images into a limited token budget for downstream multimodal reasoning and synthesis. However, existing shared-token designs are largely architecture-driven and lack an explicit criterion for what information should be preserved to simultaneously support semantic abstraction and visual detail. In this paper, we adopt a capacity-constrained perspective, viewing the shared tokenizer as a compute-bounded learner whose finite representational budget should prioritize reusable structure over hard-to-exploit high-entropy variations and redundancy. Motivated by this view, we propose \textbf{\textit{InfoTok}}, an information-regularized tokenization mechanism grounded in the Information Bottleneck (IB) principle. InfoTok explicitly controls information flow from images to shared tokens to multimodal outputs by imposing mutual-information (MI) constraints that enforce a principled trade-off between compression and task relevance, while also encouraging cross-modal consistency. Because MI is intractable for high-dimensional visual representations, we instantiate InfoTok with practical, differentiable dependence estimators, including a variational IB formulation and a Hilbert Schmidt Independence Criterion (HSIC) based alternative. Integrated into three representative unified MLLMs without introducing any additional training data, InfoTok consistently improves both image understanding and generation performance. These results support information-regularized visual tokenization as a sound basis for token learning in unified MLLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑