arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01884cs.AIcs.LG
收藏

熵引导的数据高效训练用于多模态推理奖励模型

Entropy-Guided Data-Efficient Training for Multimodal Reasoning Reward Models

School of Software, Tsinghua University(清华大学软件学院);AMAP, Alibaba Group(阿里巴巴集团AMAP)

Shidong Yang, Tongwen Huang, Hao Wen, Yong Wang, Li Chen, Xiangxiang Chu

AI总结:

本文提出熵引导训练方法,通过熵指导数据筛选和训练策略提升多模态推理奖励模型的性能。

摘要:

多模态奖励模型对于对齐多模态大语言模型与人类偏好至关重要。近期的工作已将推理能力纳入这些模型,取得了令人满意的成果。然而,训练这些模型却面临两个关键挑战:(1)偏好数据中固有的噪声,这会降低模型性能;(2)传统训练方法的低效性,这些方法忽略了样本难度的差异。在本文中,我们发现响应熵与准确性之间存在强相关性,表明熵可以作为一种可靠的、无监督的噪声和样本难度的代理。基于这一见解,我们提出了一种新的熵引导训练(EGT)方法用于多模态推理奖励模型,该方法结合了两种策略:(1)熵引导的数据整理以减轻不可靠样本的影响;(2)一种熵引导的训练策略,逐步引入更复杂的例子。在三个基准上的广泛实验表明,训练后的EGT模型在多模态奖励模型中始终优于最先进的模型。

英文摘要:

Multimodal reward models are crucial for aligning multimodal large language models with human preferences. Recent works have incorporated reasoning capabilities into these models, achieving promising results. However, training these models suffers from two critical challenges: (1) the inherent noise in preference datasets, which degrades model performance, and (2) the inefficiency of conventional training methods, which ignore the differences in sample difficulty. In this paper, we identify a strong correlation between response entropy and accuracy, indicating that entropy can serve as a reliable and unsupervised proxy for annotation noise and sample difficulty. Based on this insight, we propose a novel Entropy-Guided Training (EGT) approach for multimodal reasoning reward models, which combines two strategies: (1) entropy-guided data curation to mitigate the impact of unreliable samples, and (2) an entropy-guided training strategy that progressively introduces more complex examples. Extensive experiments across three benchmarks show that the EGT-trained model consistently outperforms state-of-the-art multimodal reward models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑