arXivDaily arXiv每日学术速递 周一至周五更新
​通知:联邦节假日调休:哥伦布日 / 原住民日(Columbus Day / Indigenous Peoples' Day),今日(13日)无更新。
arXiv 2601.03027cs.CL

通过事实性感知偏好学习减少大语言模型的幻觉

Reducing Hallucinations in LLMs via Factuality-Aware Preference Learning

Vector Institute for Artificial Intelligence(向量人工智能研究所);University of Cincinnati(辛辛那提大学);University of Calgary(卡尔加里大学)

Sindhuja Chaduvula, Ahmed Y. Radwan, Azib Farooq, Yani Ioannou, Shaina Raza

AI总结:

本文提出F-DPO方法,通过二元事实性标签和标签翻转变换提升模型事实性,减少幻觉。在七种大模型上验证效果,Qwen3-8B幻觉率降低5倍,事实性得分提升50%。

摘要:

偏好对齐方法如RLHF和DPO能提升指令遵循性,但可能强化幻觉,因其偏好判断奖励流畅性和自信而非事实正确性。我们引入F-DPO(事实性感知直接偏好优化),一种简单的DPO扩展,仅使用二元事实性标签。F-DPO(i)应用标签翻转变换,修正误排偏好对,使所选响应从未比被拒响应更不准确;(ii)添加事实性感知边缘,强调正确性差异明显的配对,当两者事实性相同时退化为标准DPO。我们通过增强DPO配对并添加二元事实性指示器和合成幻觉变体构建事实性感知偏好数据。在七种开源大语言模型(1B-14B)上,F-DPO在事实性和幻觉率方面均优于基模型和标准DPO。在Qwen3-8B上,F-DPO将幻觉率降低5倍(从0.424到0.084),事实性得分提升50%(从5.26到7.90)。F-DPO还泛化到分布外基准:在TruthfulQA上,Qwen2.5-14B在MC1准确率提升17%(0.500到0.585),在MC2准确率提升49%(0.357到0.531)。F-DPO无需辅助奖励模型、token级标注或多阶段训练。

英文摘要:

Preference alignment methods such as RLHF and Direct Preference Optimization (DPO) improve instruction following, but they can also reinforce hallucinations when preference judgments reward fluency and confidence over factual correctness. We introduce F-DPO (Factuality-aware Direct Preference Optimization), a simple extension of DPO that uses only binary factuality labels. F-DPO (i) applies a label-flipping transformation that corrects misordered preference pairs so the chosen response is never less factual than the rejected one, and (ii) adds a factuality-aware margin that emphasizes pairs with clear correctness differences, while reducing to standard DPO when both responses share the same factuality. We construct factuality-aware preference data by augmenting DPO pairs with binary factuality indicators and synthetic hallucinated variants. Across seven open-weight LLMs (1B-14B), F-DPO consistently improves factuality and reduces hallucination rates relative to both base models and standard DPO. On Qwen3-8B, F-DPO reduces hallucination rates by 5x(from 0.424 to 0.084) while improving factuality scores by 50% (from 5.26 to 7.90). F-DPO also generalizes to out-of-distribution benchmarks: on TruthfulQA, Qwen2.5-14B achieves +17% MC1 accuracy (0.500 to 0.585) and +49% MC2 accuracy (0.357 to 0.531). F-DPO requires no auxiliary reward model, token-level annotations, or multi-stage training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑