GRATH：面向大语言模型的渐进式自我求真
GRATH: Gradual Self-Truthifying for Large Language Models
AI总结：
针对大语言模型生成内容真实性不足的问题，本研究提出新型后处理方法GRATH，通过迭代生成成对真实性训练数据并结合DPO自监督优化，在不损害核心能力的前提下显著提升7B模型真实性，在TruthfulQA上超越70B模型达到SOTA。
摘要：
真实性对于大语言模型（LLM）至关重要，因为它们正越来越多地被部署到现实应用中。然而，现有大语言模型在生成真实内容方面仍存在困难，这一点从它们在TruthfulQA等基准上的一般表现就可以看出。为解决这一问题，我们提出了渐进式自我求真（GRAdual self-truTHifying，GRATH），这是一种提升大语言模型真实性的新型后处理方法。GRATH利用域外问题提示生成成对的真实性训练数据，每对数据包含一个问题及其正确答案和错误答案，随后通过直接偏好优化（DPO）优化模型，使其从答案对之间的真实性差异中学习。GRATH会迭代优化真实性数据并更新模型，以自监督的方式逐步提升模型的真实性。在实验方面，我们使用不同的7B参数大语言模型对GRATH进行评估，并与规模相近甚至更大的大语言模型在基准数据集上进行对比。结果表明，GRATH在不损害其他核心能力的前提下，有效提升了大语言模型的真实性。值得注意的是，GRATH在TruthfulQA上取得了当前最优性能，MC1准确率达54.71%，MC2准确率达69.10%，甚至超过了70B参数大语言模型的表现。
英文摘要：
Truthfulness is paramount for large language models (LLMs) as they are increasingly deployed in real-world applications. However, existing LLMs still struggle with generating truthful content, as evidenced by their modest performance on benchmarks like TruthfulQA. To address this issue, we propose GRAdual self-truTHifying (GRATH), a novel post-processing method to enhance truthfulness of LLMs. GRATH utilizes out-of-domain question prompts to generate pairwise truthfulness training data with each pair containing a question and its correct and incorrect answers, and then optimizes the model via direct preference optimization (DPO) to learn from the truthfulness difference between answer pairs. GRATH iteratively refines truthfulness data and updates the model, leading to a gradual improvement in model truthfulness in a self-supervised manner. Empirically, we evaluate GRATH using different 7B-LLMs and compare with LLMs with similar or even larger sizes on benchmark datasets. Our results show that GRATH effectively improves LLMs' truthfulness without compromising other core capabilities. Notably, GRATH achieves state-of-the-art performance on TruthfulQA, with MC1 accuracy of 54.71% and MC2 accuracy of 69.10%, which even surpass those on 70B-LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
