arXiv 2605.002242026-04-30cs.AI

TUR-DPO：基于拓扑和不确定性的直接偏好优化

TUR-DPO: Topology- and Uncertainty-Aware Direct Preference Optimization

Artificial Intelligence and Innovation Centre, University of Kurdistan, Erbil, Iraq（人工智能与创新中心，乌尔米耶大学，伊拉克）；Department of Computer Engineering, University of Kurdistan, Iran（计算机工程系，乌尔米耶大学，伊朗）；Centre for Artificial Intelligence Research and Optimisation, Torrens University Australia, Brisbane, Australia（人工智能研究与优化中心，塔伦斯大学澳大利亚，布里斯班，澳大利亚）；Research and Innovation Center, Obuda University, Budapest 1034, Hungary（研究与创新中心，奥布达大学，布达佩斯1034，匈牙利）；Center for Machine Vision and Signal Analysis (CMVS), University of Oulu, Finland（机器视觉与信号分析中心（CMVS），奥卢大学，芬兰）

Abdulhady Abas Abdullah, Fatemeh Daneshfar, Seyedali Mirjalili, Mourad Oussalah

备注：: Proceedings of the 43rd International Conference on Machine Learning (ICML 2026)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

TUR-DPO通过引入轻量级推理拓扑和结合语义忠实度、效用和拓扑质量，提升偏好对齐的稳定性与鲁棒性，同时保持训练简洁性和无需在线回滚。

摘要：

对齐大型语言模型（LLMs）与人类偏好通常通过强化学习从人类反馈（RLHF）中的近端策略优化（PPO）或更简单的直接偏好优化（DPO）实现。尽管DPO稳定且无需RL，但其将偏好视为简单的胜者-败者信号，并对易受噪声或脆弱推理链影响的偏好敏感。我们提出TUR-DPO，一种具有拓扑和不确定性意识的DPO变体，通过获取轻量级推理拓扑并结合语义忠实度、效用和拓扑质量，生成校准的不确定性信号。一个小型可学习的奖励被分解到这些信号上，并整合到一个不确定性加权的DPO目标中，该目标仍无需RL，并仅依赖固定或移动参考策略。实证结果显示，在开放的7-8B模型和涵盖数学推理、事实性问答、摘要和有益/无害对话的基准上，TUR-DPO在判断胜率、忠实度和校准方面优于DPO，同时保持训练简洁性并避免在线回滚。我们进一步观察到在多模态和长上下文设置中的一致增益，并展示了TUR-DPO在以推理为中心的任务中匹配或超过PPO，同时保持操作简洁性。

英文摘要：

Aligning large language models (LLMs) with human preferences is commonly done via reinforcement learning from human feedback (RLHF) with Proximal Policy Optimization (PPO) or, more simply, via Direct Preference Optimization (DPO). While DPO is stable and RL-free, it treats preferences as flat winner vs. loser signals and is sensitive to noisy or brittle preferences arising from fragile chains of thought. We propose TUR-DPO, a topology- and uncertainty-aware variant of DPO that rewards how answers are derived, not only what they say, by eliciting lightweight reasoning topologies and combining semantic faithfulness, utility, and topology quality into a calibrated uncertainty signal. A small learnable reward is factorized over these signals and incorporated into an uncertainty-weighted DPO objective that remains RL-free and relies only on a fixed or moving reference policy. Empirically, across open 7-8B models and benchmarks spanning mathematical reasoning, factual question answering, summarization, and helpful/harmless dialogue, TUR-DPO improves judge win-rates, faithfulness, and calibration relative to DPO while preserving training simplicity and avoiding online rollouts. We further observe consistent gains in multimodal and long-context settings, and show that TUR-DPO matches or exceeds PPO on reasoning-centric tasks while maintaining operational simplicity.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。