VaPR——面向推理的视觉-语言偏好对齐
VaPR -- Vision-language Preference alignment for Reasoning
AI总结：
针对大视觉-语言模型偏好微调中合成标注存在风格与长度噪声的问题，提出基于LLM引导编辑的难负例生成框架，构建VaPR数据集微调多系列模型，在多个基准上实现显著推理性能提升，且框架可兼容开源LLM。
摘要：
采用AI生成反馈的直接偏好优化（DPO）等偏好微调方法，在让大视觉-语言模型（LVLM）对齐人类偏好方面展现出良好前景。但现有技术忽略了合成偏好标注中普遍存在的风格偏差与长度偏差形式的噪声。为此，我们提出一种基于大语言模型（LLM）引导的响应编辑的难负例响应生成框架，该框架生成带有针对性错误的被拒响应，同时保持其与被接受响应在风格和长度上的相似性。\n利用该框架，我们构建了包含3万个高质量样本的VaPR数据集，用于微调三个LVLM系列：LLaVA-V1.5、Qwen2VL及Qwen2.5VL（参数量规模为2B到13B）。我们的VaPR模型在10个基准测试上均取得显著性能提升，其中LLaVA平均提升6.5%，Qwen2VL平均提升4.0%，Qwen2.5VL平均提升1.5%，在推理任务上提升尤为明显。缩放分析显示，性能随数据规模增长持续提升，LLaVA模型即使在较小数据规模下也能获益。此外，VaPR降低了模型在二分类问题中倾向回答“是”的问题——解决了LLaVA等LVLM的常见失效模式。最后，我们证明该框架可推广到开源LLM作为编辑器，基于VaPR-OS训练的模型性能达到了使用GPT-4o合成的VaPR训练模型的约99%。我们的数据、模型和代码可在项目页面https://vap-r.github.io获取。
英文摘要：
Preference finetuning methods like Direct Preference Optimization (DPO) with AI-generated feedback have shown promise in aligning Large Vision-Language Models (LVLMs) with human preferences. However, existing techniques overlook the prevalence of noise in synthetic preference annotations in the form of stylistic and length biases. To this end, we introduce a hard-negative response generation framework based on LLM-guided response editing, that produces rejected responses with targeted errors, maintaining stylistic and length similarity to the accepted ones. Using this framework, we develop the VaPR dataset, comprising 30K high-quality samples, to finetune three LVLM families: LLaVA-V1.5, Qwen2VL & Qwen2.5VL (2B-13B sizes). Our VaPR models deliver significant performance improvements across ten benchmarks, achieving average gains of 6.5% (LLaVA), 4.0% (Qwen2VL), and 1.5% (Qwen2.5VL), with notable improvements on reasoning tasks. A scaling analysis shows that performance consistently improves with data size, with LLaVA models benefiting even at smaller scales. Moreover, VaPR reduces the tendency to answer "Yes" in binary questions - addressing a common failure mode in LVLMs like LLaVA. Lastly, we show that the framework generalizes to open-source LLMs as editors, with models trained on VaPR-OS achieving ~99% of the performance of models trained on \name, which is synthesized using GPT-4o. Our data, models, and code can be found on the project page https://vap-r.github.io
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
