arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.20520cs.CV
收藏

上下文标记是锚点:从信息流的角度理解dMLLMs中的重复诅咒

Context Tokens are Anchors: Understanding the Repetition Curse in dMLLMs from an Information Flow Perspective

SJTU(上海交通大学);CASIA(中国科学院自动化研究所);NTU(国立台湾大学);Meitu (China) Limited(美图公司);XMUT(新疆大学)

Qiyan Zhao, Xiaofeng Zhang, Shuochen Chang, Qianyu Chen, Xiaosong Yuan, Xuhang Chen, Luoqi Liu, Jiajun Zhang, Xu-Yao Zhang, Da-Han Wang

AI总结:

CoTA通过增强上下文标记注意力和引入惩罚项缓解dMLLMs中的重复问题,提升解码性能。

摘要:

近期的基于扩散的多模态大语言模型(dMLLMs)由于推理延迟高,因此依赖缓存技术来加速解码。然而,缓存机制的应用往往引入了不希望的重复文本生成,这种现象我们称之为

英文摘要:

Recent diffusion-based Multimodal Large Language Models (dMLLMs) suffer from high inference latency and therefore rely on caching techniques to accelerate decoding. However, the application of cache mechanisms often introduces undesirable repetitive text generation, a phenomenon we term the \textbf{Repeat Curse}. To better investigate underlying mechanism behind this issue, we analyze repetition generation through the lens of information flow. Our work reveals three key findings: (1) context tokens aggregate semantic information as anchors and guide the final predictions; (2) as information propagates across layers, the entropy of context tokens converges in deeper layers, reflecting the model's growing prediction certainty; (3) Repetition is typically linked to disruptions in the information flow of context tokens and to the inability of their entropy to converge in deeper layers. Based on these insights, we present \textbf{CoTA}, a plug-and-play method for mitigating repetition. CoTA enhances the attention of context tokens to preserve intrinsic information flow patterns, while introducing a penalty term to the confidence score during decoding to avoid outputs driven by uncertain context tokens. With extensive experiments, CoTA demonstrates significant effectiveness in alleviating repetition and achieves consistent performance improvements on general tasks. Code is available at https://github.com/ErikZ719/CoTA

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑