Omni-RRM：通过基于自动评分标准的偏好合成推进全模态奖励建模
Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis
AI总结：
研究针对多模态大语言模型对齐难题，提出全模态基于评分标准的奖励模型Omni-RRM。通过自动评分标准偏好合成构建数据集，用特定训练方案提升奖励辨别力，在多模态基准测试中达最优精度，还能指导选择与迁移，提升模型对齐能力。
摘要：
多模态大语言模型（MLLMs）因现有奖励模型（RMs）的局限性而在对齐方面面临困难，这些模型主要以视觉为中心，依赖昂贵的人工标签，且提供的不透明标量分数无法捕捉细微推理，导致对齐脆弱。我们提出了Omni-RRM，一种全模态、基于评分标准的奖励模型，可生成跨文本、图像、视频和音频的多维奖励信号。为克服多维度推理中以人类为中心评估的高成本和内在不一致性，我们引入了Omni-Preference，一个通过基于自动评分标准的偏好合成构建的高质量数据集。在这个流程中，教师模型将原始偏好协调为明确的理由，确保合成监督既高保真又可解释。Omni-RRM使用渐进式SFT + GRPO方案进行训练，经过专门优化以增强对低边际、硬偏好对的奖励辨别能力。它在视频（ShareGPT-Video上80.2%）和音频基准测试（Audio-HH-RLHF上66.8%，TA2T上65.0%）上达到了当前最优精度，五个基准测试的总体准确率为70.4%，相对于其主干模型有+17.0%的相对增益。此外，Omni-RRM有效地指导了N选优选择，并在仅文本对齐方面表现出强大的迁移能力。所有资源，包括数据集、训练和推理代码以及模型检查点，都可在该https URL获取。
英文摘要：
Multimodal large language models (MLLMs) struggle with alignment due to the limitations of existing reward models (RMs), which are predominantly vision-centric, dependent on costly human labels, and provide opaque scalar scores that fail to capture nuanced reasoning, leading to brittle alignment. We present Omni-RRM, an \textbf{Omni}-modal \textbf{R}ubric-grounded \textbf{R}eward \textbf{M}odel that generates multi-dimensional reward signals across text, image, video, and audio. To overcome the high cost and inherent inconsistency of human-centric evaluation in multi-dimensional reasoning, we introduce \textbf{Omni-Preference}, a high-quality dataset constructed via automatic rubric-grounded preference synthesis. In this pipeline, teacher models reconcile raw preferences into explicit justifications, ensuring that the synthesized supervision is both high-fidelity and interpretable. Omni-RRM is trained using a progressive SFT + GRPO regimen, specifically optimized to sharpen reward discrimination on low-margin, hard preference pairs. It achieves state-of-the-art accuracy on video (80.2\% on ShareGPT-Video) and audio benchmarks (66.8\% on Audio-HH-RLHF and 65.0\% on TA2T), yielding a five-benchmark Overall accuracy of 70.4\% and a +17.0\% relative gain over its backbone. Furthermore, Omni-RRM effectively guides Best-of-$N$ selection and exhibits robust transfer to text-only alignment. All resources, including the dataset, training and inference code, and model checkpoints are available at https://tmfk418.github.io/Omni-RRM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
