不熟悉的微调样本控制语言模型如何产生幻觉
Unfamiliar Finetuning Examples Control How Language Models Hallucinate
AI总结：
本文发现微调数据中超出基础模型知识范围的不熟悉样本会塑造语言模型的幻觉，通过策略性修改这些样本的监督方式可控制模型对未知查询的响应，从而提升长文本生成的事实性。
摘要：
大型语言模型在面对不熟悉的查询时会产生幻觉，但控制模型如何产生幻觉的潜在机制尚未完全被理解。在这项工作中，我们发现模型微调数据中的不熟悉样本——即引入基础模型知识范围之外概念的样本——在塑造这些错误中起着关键作用。具体而言，我们发现LLM的幻觉预测倾向于映射其不熟悉微调样本相关的响应。这表明，通过修改不熟悉微调样本的监督方式，我们可以影响模型对不熟悉查询的响应（例如，说“我不知道”）。我们在涉及SFT、RL和奖励模型微调的TriviaQA和MMLU上的一系列对照实验中实证验证了这一观察结果。我们的工作进一步研究了用于提高长文本模型生成事实性的RL微调策略。我们发现，虽然奖励模型产生的幻觉会显著削弱RL事实性微调的有效性，但策略性地控制奖励模型如何产生幻觉可以最小化这些负面影响。利用我们先前关于控制幻觉的观察，我们提出了一种学习更可靠奖励模型的方法，并表明它们在长文本传记和书籍/电影情节生成任务中提高了RL事实性微调的有效性。
英文摘要：
Large language models are known to hallucinate when faced with unfamiliar queries, but the underlying mechanism that govern how models hallucinate are not yet fully understood. In this work, we find that unfamiliar examples in the models' finetuning data -- those that introduce concepts beyond the base model's scope of knowledge -- are crucial in shaping these errors. In particular, we find that an LLM's hallucinated predictions tend to mirror the responses associated with its unfamiliar finetuning examples. This suggests that by modifying how unfamiliar finetuning examples are supervised, we can influence a model's responses to unfamiliar queries (e.g., say ``I don't know''). We empirically validate this observation in a series of controlled experiments involving SFT, RL, and reward model finetuning on TriviaQA and MMLU. Our work further investigates RL finetuning strategies for improving the factuality of long-form model generations. We find that, while hallucinations from the reward model can significantly undermine the effectiveness of RL factuality finetuning, strategically controlling how reward models hallucinate can minimize these negative effects. Leveraging our previous observations on controlling hallucinations, we propose an approach for learning more reliable reward models, and show that they improve the efficacy of RL factuality finetuning in long-form biography and book/movie plot generation tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
