S2H-DPO：面向视觉-语言模型的硬度感知偏好优化
S2H-DPO: Hardness-Aware Preference Optimization for Vision-Language Models
AI总结：
本文提出S2H-DPO框架，通过三级层次推理构建多图象偏好数据，提升多图象推理性能，同时保持单图象推理能力，推动视觉偏好对齐的前沿发展。
摘要：
视觉-语言模型（VLMs）在单图象理解方面取得了显著进展，但多图象推理仍面临挑战。本文识别出现有多图象对齐方法的关键能力缺口：当前方法主要依赖预指定图象索引进行局部推理（如'看图象3'），忽略了全局视觉搜索和自主跨图象比较的核心技能。为解决这一限制，我们引入简单到困难（S2H）学习框架，系统构建跨三个层次推理级别的多图象偏好数据：（1）单图象局部推理，（2）多图象局部比较，（3）全局视觉搜索。与以往依赖模型特定属性（如幻觉或注意力启发式）生成偏好对的方法不同，我们的方法利用提示驱动的复杂性生成适用于不同模型的所选/拒绝对。通过在LLaVA和Qwen-VL模型上的广泛评估，我们证明了我们的多样化多图象推理数据显著提升了多图象推理性能，在多个基准上优于基线方法。重要的是，我们的方法在保持单图象推理性能的同时，增强了多图象理解能力，从而推动了视觉偏好对齐的前沿发展。
英文摘要：
Vision-Language Models (VLMs) have demonstrated remarkable progress in single-image understanding, yet effective reasoning across multiple images remains challenging. We identify a critical capability gap in existing multi-image alignment approaches: current methods focus primarily on localized reasoning with pre-specified image indices (``Look at Image 3 and...''), bypassing the essential skills of global visual search and autonomous cross-image comparison. To address this limitation, we introduce a Simple-to-Hard (S2H) learning framework that systematically constructs multi-image preference data across three hierarchical reasoning levels requiring an increasing level of capabilities: (1) single-image localized reasoning, (2) multi-image localized comparison, and (3) global visual search. Unlike prior work that relies on model-specific attributes, such as hallucinations or attention heuristics, to generate preference pairs, our approach leverages prompt-driven complexity to create chosen/rejected pairs that are applicable across different models. Through extensive evaluations on LLaVA and Qwen-VL models, we show that our diverse multi-image reasoning data significantly enhances multi-image reasoning performance, yielding significant improvements over baseline methods across benchmarks. Importantly, our approach maintains strong single-image reasoning performance while simultaneously strengthening multi-image understanding capabilities, thus advancing the state of the art for holistic visual preference alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
