arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.04780cs.CV
收藏

CLEAR: 解锁统一多模态模型中退化图像理解的生成潜力

CLEAR: Unlocking Generative Potential for Degraded Image Understanding in Unified Multimodal Models

Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Baidu Inc.(百度公司)

Xiangzhao Hao, Zefeng Zhang, Zhenyu Zhang, Linhao Yu, Yao Chen, Yiqian Zhang, Haiyun Guo, Shuohuan Wang, Yu Sun

AI总结:

CLEAR通过三个步骤提升统一多模态模型在退化图像中的理解能力,包括监督微调、潜在表示桥梁和交错GRPO,增强模型在退化输入下的鲁棒性。

摘要:

图像退化(如模糊、噪声、压缩和光照不良)严重削弱了现实场景中的多模态理解。统一多模态模型将理解和生成结合在单一架构中,其生成路径可建模被退化破坏的细粒度视觉结构。然而,这些模型未能利用其生成能力处理退化输入。我们发现这一断层源于两个因素:现有训练方案从未要求模型在推理中调用生成,且标准解码-重编码路径不支持有效联合优化。我们提出CLEAR框架,通过三个渐进步骤连接两种能力:(1)在退化感知数据集上进行监督微调,建立生成后回答的推理模式;(2)潜在表示桥梁,用直接可优化的连接替代解码-重编码绕道;(3)交错GRPO,一种强化学习方法,通过答案正确性奖励联合优化文本推理和视觉生成。我们构建了MMD-Bench,涵盖六个标准多模态基准的三个退化严重程度级别。实验表明,CLEAR显著提高了在退化输入上的鲁棒性,同时保持干净图像性能。我们的分析进一步表明,去除像素级重建监督会导致中间视觉状态具有更高的感知质量,表明任务驱动优化和视觉质量自然对齐。

英文摘要:

Image degradation from blur, noise, compression, and poor illumination severely undermines multimodal understanding in real-world settings. Unified multimodal models that combine understanding and generation within a single architecture are a natural fit for this challenge, as their generative pathway can model the fine-grained visual structure that degradation destroys. Yet these models fail to leverage their own generative capacity on degraded inputs. We trace this disconnect to two compounding factors: existing training regimes never ask the model to invoke generation during reasoning, and the standard decode-reencode pathway does not support effective joint optimization. We present CLEAR, a framework that connects the two capabilities through three progressive steps: (1) supervised fine-tuning on a degradation-aware dataset to establish the generate-then-answer reasoning pattern; (2) a Latent Representation Bridge that replaces the decode-reencode detour with a direct, optimizable connection between generation and reasoning; (3) Interleaved GRPO, a reinforcement learning method that jointly optimizes text reasoning and visual generation under answer-correctness rewards. We construct MMD-Bench, covering three degradation severity levels across six standard multimodal benchmarks. Experiments show that CLEAR substantially improves robustness on degraded inputs while preserving clean-image performance. Our analysis further reveals that removing pixel-level reconstruction supervision leads to intermediate visual states with higher perceptual quality, suggesting that task-driven optimization and visual quality are naturally aligned.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑