面向事实性的自对齐：通过自评估缓解 LLM 幻觉
Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation
AI总结：
本文提出面向事实性的 LLM 自对齐方法，通过 Self-Eval 与 SK-Tuning 生成自评估偏好信号并以 DPO 微调，显著提升 Llama 模型在 TruthfulQA 和 BioGEN 上的事实准确性。
摘要：
尽管大型语言模型（LLM）展现出越来越接近人类的能力，但它们仍常受到事实不准确（即“幻觉”）的困扰，即使其内部已具备相关知识也是如此。为解决这些幻觉，现有方法通常需要高质量的人工事实性标注。本文探索面向事实性的自对齐，利用 LLM 的自评估能力提供训练信号，引导模型趋向事实性。具体而言，我们引入 Self-Eval 这一自评估组件，提示 LLM 仅依据内部知识验证其自身生成回答的事实性。此外，我们设计 Self-Knowledge Tuning（SK-Tuning），通过改进模型的置信度估计与校准来增强 LLM 的自评估能力。随后，我们利用这些自标注回答，通过 Direct Preference Optimization 算法对模型进行微调。结果表明，在 TruthfulQA 和 BioGEN 上的三项关键知识密集型任务中，所提出的自对齐方法显著提升了 Llama 系列模型的事实准确性。
英文摘要：
Despite showing increasingly human-like abilities, large language models (LLMs) often struggle with factual inaccuracies, i.e. "hallucinations", even when they hold relevant knowledge. To address these hallucinations, current approaches typically necessitate high-quality human factuality annotations. In this work, we explore Self-Alignment for Factuality, where we leverage the self-evaluation capability of an LLM to provide training signals that steer the model towards factuality. Specifically, we incorporate Self-Eval, a self-evaluation component, to prompt an LLM to validate the factuality of its own generated responses solely based on its internal knowledge. Additionally, we design Self-Knowledge Tuning (SK-Tuning) to augment the LLM's self-evaluation ability by improving the model's confidence estimation and calibration. We then utilize these self-annotated responses to fine-tune the model via Direct Preference Optimization algorithm. We show that the proposed self-alignment approach substantially enhances factual accuracy over Llama family models across three key knowledge-intensive tasks on TruthfulQA and BioGEN.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
