DGPO: 超越成对偏好与方向一致的组级优化
DGPO: Beyond Pairwise Preferences with Directional Consistent Groupwise Optimization
AI总结：
本文提出DGPO框架，通过组级监督信号和多候选比较建模方向一致性，提升偏好优化的准确性和多样性，实验显示在多个数据集上平均提升3.6%。
摘要：
尽管大型语言模型（LLMs）取得了显著进展，当前偏好优化方法仍难以在保持推理多样性的同时实现方向一致性。为解决这一限制，我们提出了方向-组级偏好优化（DGPO），一个轻量级框架，通过组级聚合监督信号并利用多候选比较显式建模方向感知对齐。DGPO将正反问题-答案实例组织成结构化集合，并优化一个基于边界的似然目标，将一致的推理路径与不一致的替代方案分开。这种组级建模比成对目标捕获更丰富的相对信息，并在多样化的推理路径上强化一致性。实验结果表明，我们构建的反向数据在五个基准上平均提升3.2%，而DGPO在多个数据集和模型家族上进一步实现一致的提升，平均准确率提升高达3.6%。
英文摘要：
Although Large Language Models (LLMs) have made remarkable progress, current preference optimization methods still struggle to align directional consistency while preserving reasoning diversity. To address this limitation, we propose Directional-Groupwise Preference Optimization (DGPO), a lightweight framework that aggregates supervision signals at the group level and explicitly models direction-aware alignment through multi-candidate comparisons. DGPO organizes forward and reverse question-answer instances into structured sets and optimizes a margin-based likelihood objective that separates coherent reasoning paths from inconsistent alternatives. This group-wise formulation captures richer relative information than pairwise objectives and reinforces consistency across diverse reasoning pathways. Empirical results show that our constructed reverse data yields a 3.2% average improvement across five benchmarks, while DGPO further delivers consistent gains across multiple datasets and model families, achieving average accuracy improvements of up to 3.6%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
