在自进化大型多模态模型中更加关注视觉标记
Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models
AI总结：
提出VISE框架，通过几何不变性和语义不变性奖励直接正则化视觉条件策略，解决自进化LMMs中视觉欠条件问题，在无监督设置下提升视觉语言理解。
摘要：
近期，自进化大型多模态模型（LMMs）因在纯无监督设置下提升视觉推理能力而受到关注。然而，现有自进化LMMs中的多角色自博弈和自一致性奖励方案优化了答案一致性，却未确保解码器关注视觉内容，而是依赖统计语言先验产生自一致输出。这导致了一种我们称之为视觉欠条件的持续失败模式，即解码器在生成过程中依赖语言先验而非图像，表现为对视觉标记的关注不足。因此，当前自进化LMMs在图像描述和视觉问答等视觉-语言理解任务上表现不佳。为解决此问题，我们提出VISE（视觉不变性自进化），一种纯无监督的自进化框架，通过两种互补的基于不变性的奖励直接正则化模型的视觉条件策略：几何不变性奖励在已知变换下强制空间一致性，语义不变性奖励通过要求模型在预测区域被扰动时识别证据缺失来惩罚无证据生成。VISE在单一模型内运行，无需专家角色、外部奖励模型或标注，并在原始无标签图像上训练。在18个基准上的实验证明了我们方法的有效性。以Qwen3-VL-2B为基础模型，VISE在COCO上获得+16.85 CIDEr，在TextCaps上获得+19.66 CIDEr，将物体幻觉降低5.0 Chair-I点，并在四个模型系列和规模上泛化。我们的代码和模型见此https URL。
英文摘要：
Recently, self-evolving large multimodal models (LMMs) have received attention for improving visual reasoning in a purely unsupervised setting. However, multi-role self-play and self-consistency reward schemes in existing self-evolving LMMs optimize answer agreement without ensuring the decoder attends to visual content, relying instead on statistical language priors to produce self consistent outputs. This leads to a persistent failure mode we term visual under-conditioning, where the decoder relies on language priors rather than the image during generation, manifesting as insufficient attention to visual tokens. As a result, current self-evolving LMMs struggle on vision--language understanding tasks such as image captioning and visual question answering. To address this, we propose VISE (Visual Invariance Self-Evolution), a purely unsupervised self-evolving framework that directly regularizes the model's visual conditioning policy through two complementary invariance-based rewards: a geometric invariance reward that enforces spatial consistency under known transformations, and a semantic invariance reward that penalizes evidence-agnostic generation by requiring the model to recognize the absence of evidence when predicted regions are perturbed. VISE operates within a single model without specialist roles, external reward models, or annotations, and is trained on raw unlabeled images. Experiments on 18 benchmarks demonstrate the efficacy of our approach. Using Qwen3-VL-2B as the base model, VISE achieves gains of $+16.85$ CIDEr on COCO and $+19.66$ CIDEr on TextCaps, reduces object hallucination by $5.0$ Chair-I points, and generalizes across four model families and scales. Our code and models are available at https://mbzuai-oryx.github.io/VISE
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
