从字幕到奖励（CAREVL）：利用大语言模型专家增强大视觉-语言模型中的奖励建模
From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Models
AI总结：
本文提出CAREVL方法，通过文本奖励模型利用图像字幕筛选高置信度数据，并用低置信度数据生成可靠偏好配对，从而提升大视觉-语言模型的奖励建模性能。
摘要：
由于缺乏无需人工标注的细粒度、高质量和多模态偏好数据，将大视觉-语言模型（LVLMs）与人类偏好对齐具有挑战性。现有依赖直接蒸馏的方法往往难以处理低置信度数据，导致性能次优。为解决这一问题，我们提出CAREVL，一种通过可靠地同时利用高置信度和低置信度数据进行偏好奖励建模的新方法。首先，一组辅助专家模型（文本奖励模型）创新性地利用图像字幕作为弱监督信号来筛选高置信度数据。随后，高置信度数据被用于微调LVLM。其次，低置信度数据被用于通过微调后的LVLM生成多样化的偏好样本。这些样本随后被评分和筛选，以构建可靠的选定-拒绝配对用于进一步训练。CAREVL在VL-RewardBench和MLLM-as-a-Judge基准上相较于传统基于蒸馏的方法实现了性能提升，证明了其有效性。代码将很快发布。
英文摘要：
Aligning large vision-language models (LVLMs) with human preferences is challenging due to the scarcity of fine-grained, high-quality, and multimodal preference data without human annotations. Existing methods relying on direct distillation often struggle with low-confidence data, leading to suboptimal performance. To address this, we propose CAREVL, a novel method for preference reward modeling by reliably using both high- and low-confidence data. First, a cluster of auxiliary expert models (textual reward models) innovatively leverages image captions as weak supervision signals to filter high-confidence data. The high-confidence data are then used to fine-tune the LVLM. Second, low-confidence data are used to generate diverse preference samples using the fine-tuned LVLM. These samples are then scored and selected to construct reliable chosen-rejected pairs for further training. CAREVL achieves performance improvements over traditional distillation-based methods on VL-RewardBench and MLLM-as-a-Judge benchmark, demonstrating its effectiveness. The code will be released soon.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
