超越欠对齐：面向大语言模型的原子偏好增强事实性微调
Beyond Under-Alignment: Atomic Preference Enhanced Factuality Tuning for Large Language Models
AI总结：
本文提出APEFT框架，通过原子偏好增强事实性微调，解决LLMs在分布偏移下因欠对齐导致的幻觉问题，在域内外数据集上平均提升3.45%性能。
摘要：
大型语言模型（LLMs）取得了显著成功，但仍倾向于生成事实错误的响应，这一现象被称为幻觉。近期趋势是利用偏好学习微调模型以符合事实性。然而，现有工作主要在域内（ID）数据集上评估微调模型，域外（OOD）数据集上的事实性仍被探讨不足。本文对各种偏好学习算法微调的不同模型的事实性进行了全面评估，表明它们在OOD数据集上的性能要么增加极少，要么下降。随后，通过分析微调前后模型的token分布偏移，我们揭示了模型在分布偏移下未能保持事实性的主要原因是欠对齐，而非过对齐。最后，我们提出了APEFT（原子偏好增强事实性微调），这是一个在单个事实粒度上增强模型事实性意识的框架。大量实验表明，APEFT在ID和OOD数据集上平均提升模型性能3.45%，效果显著。
英文摘要：
Large language models (LLMs) have achieved remarkable success but still tend to generate factually erroneous responses, a phenomenon known as hallucination. A recent trend is to use preference learning to fine-tune models to align with factuality. However, existing work primarily evaluates fine-tuned models on in-domain (ID) datasets and the factuality on out-of-domain (OOD) datasets remains underexplored. In this paper, we conduct a comprehensive evaluation of the factuality of different models tuned by various preference learning algorithms and demonstrate that their performance on OOD datasets either increases minimally or decreases. Subsequently, we reveal that the main cause of model's failure to uphold factuality under a distribution shift is \textbf{under-alignment}, rather than \textbf{over-alignment}, by analyzing the token distribution shift of the models before and after tuning. Finally, we propose \textbf{APEFT} (\textbf{A}tomic \textbf{P}reference \textbf{E}nhanced \textbf{F}actuality \textbf{T}uning), a framework that enhances model's awareness of factuality at the granularity of individual facts. Extensive experiments demonstrate that APEFT improves model performance by an average of $\boldsymbol{3.45\%}$ on both ID and OOD datasets, which is highly effective.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
