arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.21743cs.CV
收藏

通过难度感知分组归一化增强多模态大语言模型推理

Enhancing Multi-Modal LLMs Reasoning via Difficulty-Aware Group Normalization

University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学)

Jinghan Li, Junfeng Fang, Jinda Lu, Yuan Wang, Xiaoyan Guo, Tianyu Zhang, Xiang Wang, Xiangnan He

AI总结:

本文提出难度感知分组归一化方法,通过感知复杂度和推理不确定性表征样本,提升多模态大语言模型的推理稳定性与性能。

摘要:

可验证奖励的强化学习(RLVR)和组相对策略优化(GRPO)显著提升了大语言模型的推理能力。然而,将这些方法扩展到多模态设置时面临关键挑战:基于标准差的归一化不稳定,容易受到接近正负奖励的极端样本的扭曲。与纯文本LLM不同,多模态模型对这种扭曲特别敏感,因为感知和推理错误都会影响其响应。为此,我们通过感知复杂度(通过视觉熵测量)和推理不确定性(通过模型置信度捕捉)来表征每个样本。基于此表征,我们提出了难度感知分组归一化(Durian),通过难度级别重新分组样本并在每个组内共享标准差。我们的方法在保持GRPO组内区别的同时消除了对极端案例的敏感性,在多个多模态推理基准上实现了显著的性能提升。

英文摘要:

Reinforcement Learning with Verifiable Rewards (RLVR) and Group Relative Policy Optimization (GRPO) have significantly advanced the reasoning capabilities of large language models. Extending these methods to multimodal settings, however, faces a critical challenge: the instability of std-based normalization, which is easily distorted by extreme samples with nearly positive or negative rewards. Unlike pure-text LLMs, multimodal models are particularly sensitive to such distortions, as both perceptual and reasoning errors influence their responses. To address this, we characterize each sample by its difficulty, defined through perceptual complexity (measured via visual entropy) and reasoning uncertainty (captured by model confidence). Building on this characterization, we propose difficulty-aware group normalization (Durian), which re-groups samples by difficulty levels and shares the std within each group. Our approach preserves GRPO's intra-group distinctions while eliminating sensitivity to extreme cases, yielding significant performance gains across multiple multimodal reasoning benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑