语言模型的事实性微调
Fine-tuning Language Models for Factuality
AI总结：
本文提出利用自动生成的事实性偏好排序微调语言模型，无需人工标注，显著提升Llama-2在开放生成任务中的事实性，错误率降低达58%。
摘要：
大型预训练语言模型（LLMs）的流畅性和创造性使其得到广泛应用，有时甚至被用作传统搜索引擎的替代品。然而，语言模型容易产生令人信服但事实不准确的表述，这通常被称为“幻觉”。这些错误可能无意中传播错误信息或有害地加深误解。此外，对模型响应进行人工事实核查是一个耗时的过程，使得人工事实性标注成本高昂。在这项工作中，我们在无需人工标注的情况下微调语言模型以提高其事实性，并针对比以往工作更开放的生成场景。我们利用自然语言处理领域的两项关键近期创新来实现这一目标。首先，近期多项工作提出了通过衡量与外部知识库的一致性，或仅使用大型模型的置信度分数来评判开放文本事实性的方法。其次，直接偏好优化算法使得语言模型能够在监督模仿之外的目标上进行直接微调，利用对可能模型响应的偏好排序。我们表明，从自动生成的事实性偏好排序中学习——这些排序通过现有检索系统或我们新颖的无检索方法生成——显著提高了Llama-2在未见主题上生成内容的事实性（正确生成主张的百分比），相比针对事实性的RLHF或解码策略。在7B规模下，与Llama-2-chat相比，在生成传记和回答医学问题时，我们观察到事实错误率分别降低了58%和40%。
英文摘要：
The fluency and creativity of large pre-trained language models (LLMs) have led to their widespread use, sometimes even as a replacement for traditional search engines. Yet language models are prone to making convincing but factually inaccurate claims, often referred to as 'hallucinations.' These errors can inadvertently spread misinformation or harmfully perpetuate misconceptions. Further, manual fact-checking of model responses is a time-consuming process, making human factuality labels expensive to acquire. In this work, we fine-tune language models to be more factual, without human labeling and targeting more open-ended generation settings than past work. We leverage two key recent innovations in NLP to do so. First, several recent works have proposed methods for judging the factuality of open-ended text by measuring consistency with an external knowledge base or simply a large model's confidence scores. Second, the direct preference optimization algorithm enables straightforward fine-tuning of language models on objectives other than supervised imitation, using a preference ranking over possible model responses. We show that learning from automatically generated factuality preference rankings, generated either through existing retrieval systems or our novel retrieval-free approach, significantly improves the factuality (percent of generated claims that are correct) of Llama-2 on held-out topics compared with RLHF or decoding strategies targeted at factuality. At 7B scale, compared to Llama-2-chat, we observe 58% and 40% reduction in factual error rate when generating biographies and answering medical questions, respectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
