Mask-DPO：大型语言模型的可泛化细粒度事实性对齐
Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMs
AI总结：
本文提出Mask-DPO方法，利用句子级事实性作为掩码信号优化DPO训练，有效消除非偏好样本中事实内容的惩罚，显著提升了LLM在域内外数据集上的回复事实性并展现出优异的泛化能力。
摘要：
大型语言模型（LLMs）在各个领域充当AI助手时会产生幻觉（即不忠实或无意义的信息）。由于幻觉在LLM的回复中总是与真实内容相伴，以往进行回复级偏好学习的事实性对齐方法在训练过程中不可避免地引入了噪声。因此，本文提出了一种基于直接偏好优化（DPO）的细粒度事实性对齐方法，称为Mask-DPO。Mask-DPO将句子级事实性作为掩码信号，仅从偏好样本中事实正确的句子进行学习，并防止对非偏好样本中的事实内容进行惩罚，从而解决了偏好学习中的歧义问题。大量实验结果表明，Mask-DPO能显著提升LLM对域内和域外数据集问题的回复事实性，尽管这些问题及其相关主题在训练期间未被见过。仅在ANAH训练集上进行训练，Llama3.1-8B-Instruct在ANAH测试集上的得分从49.19%提升至77.53%，甚至超过了Llama3.1-70B-Instruct的得分（53.44%），同时其在域外Biography数据集上的FactScore也从30.29%提升至39.39%。我们进一步使用不同的训练样本缩放策略研究了Mask-DPO的泛化特性，发现增加数据集中的主题数量比增加问题数量更有效。我们基于这一现象提供了关于事实性对齐对LLM作用机制的假设，并进行了概念验证实验来验证它。我们希望该方法和这些发现能为未来关于扩展事实性对齐的研究铺平道路。
英文摘要：
Large language models (LLMs) exhibit hallucinations (i.e., unfaithful or nonsensical information) when serving as AI assistants in various domains. Since hallucinations always come with truthful content in the LLM responses, previous factuality alignment methods that conduct response-level preference learning inevitably introduced noises during training. Therefore, this paper proposes a fine-grained factuality alignment method based on Direct Preference Optimization (DPO), called Mask-DPO. Incorporating sentence-level factuality as mask signals, Mask-DPO only learns from factually correct sentences in the preferred samples and prevents the penalty on factual contents in the not preferred samples, which resolves the ambiguity in the preference learning. Extensive experimental results demonstrate that Mask-DPO can significantly improve the factuality of LLMs responses to questions from both in-domain and out-of-domain datasets, although these questions and their corresponding topics are unseen during training. Only trained on the ANAH train set, the score of Llama3.1-8B-Instruct on the ANAH test set is improved from 49.19% to 77.53%, even surpassing the score of Llama3.1-70B-Instruct (53.44%), while its FactScore on the out-of-domain Biography dataset is also improved from 30.29% to 39.39%. We further study the generalization property of Mask-DPO using different training sample scaling strategies and find that scaling the number of topics in the dataset is more effective than the number of questions. We provide a hypothesis of what factual alignment is doing with LLMs, on the implication of this phenomenon, and conduct proof-of-concept experiments to verify it. We hope the method and the findings pave the way for future research on scaling factuality alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
