arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.04145cs.LGcs.CLcs.MM
收藏

多模态过程奖励模型中的训练数据效率

Training Data Efficiency in Multimodal Process Reward Models

Washington University in St. Louis(华盛顿大学圣路易斯分校);Singapore University of Technology(新加坡科技设计大学);University of Virginia(弗吉尼亚大学)

Jinyuan Li, Chengsong Huang, Langlin Huang, Shaoyang Xu, Haolin Liu, Wenxuan Zhang, Jiaxin Huang

AI总结:

本文提出BIS方法,通过优化标签混合与可靠性,提升多模态过程奖励模型训练的数据效率,仅用10%训练数据即可达到全数据性能。

摘要:

多模态过程奖励模型(MPRMs)是MLLMs中视觉推理的步级监督的核心。训练MPRMs通常需要大规模的蒙特卡洛(MC)标注语料库,这会带来显著的训练成本。本文研究了MPRMs训练的数据效率。我们的初步实验表明,在随机子采样训练数据时,MPRMs训练会迅速饱和,表明现有MC标注语料库中存在大量冗余。为了解释这一点,我们建立了一个理论框架,揭示了信息梯度更新依赖于两个因素:正/负步骤的标签混合和标签可靠性(正步骤的平均MC分数)。基于这些见解,我们提出了平衡信息分数(BIS),它在回放级别利用现有的MC信号,优先考虑混合和可靠性,而无需增加任何成本。在VisualProcessBench上,使用两种主干(InternVL2.5-8B和Qwen2.5-VL-7B),BIS选择的子集在小比例下一致匹配甚至超过了全数据性能。值得注意的是,BIS子集仅使用10%的训练数据即可达到全数据性能,比随机子采样提高了4.1%。

英文摘要:

Multimodal Process Reward Models (MPRMs) are central to step-level supervision for visual reasoning in MLLMs. Training MPRMs typically requires large-scale Monte Carlo (MC)-annotated corpora, incurring substantial training cost. This paper studies the data efficiency for MPRM training. Our preliminary experiments reveal that MPRM training quickly saturates under random subsampling of the training data, indicating substantial redundancy within existing MC-annotated corpora. To explain this, we formalize a theoretical framework and reveal that informative gradient updates depend on two factors: label mixtures of positive/negative steps and label reliability (average MC scores of positive steps). Guided by these insights, we propose the Balanced-Information Score (BIS), which prioritizes both mixture and reliability based on existing MC signals at the rollout level, without incurring any additional cost. Across two backbones (InternVL2.5-8B and Qwen2.5-VL-7B) on VisualProcessBench, BIS-selected subsets consistently match and even surpass the full-data performance at small fractions. Notably, the BIS subset reaches full-data performance using only 10% of the training data, improving over random subsampling by a relative 4.1%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑