TruthRL: 通过强化学习激励诚实的LLM
TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning
AI总结：
提出TruthRL框架，使用GRPO和三值奖励直接优化LLM的诚实性，减少幻觉并允许不确定时弃权，在知识密集型基准上显著提升诚实性。
摘要：
虽然大型语言模型（LLM）在事实性问题回答上表现出色，但它们仍然容易产生幻觉和不真实的回答，特别是当任务需要其参数知识之外的信息时。事实上，诚实性需要的不仅仅是准确性——模型还必须识别不确定性，并在不确定时弃权以避免幻觉。这对现有方法提出了根本性挑战：优化准确性的方法往往会放大幻觉，而鼓励弃权的方法可能变得过于保守，牺牲正确答案。两种极端最终都损害了诚实性。在这项工作中，我们提出了TruthRL，一个通用的强化学习（RL）框架，直接优化LLM的诚实性。具体来说，我们使用GRPO实现TruthRL，并采用一个简单而有效的三值奖励，区分正确答案、幻觉和弃权。它激励模型不仅通过提供正确回答来减少幻觉，还通过在不确定时启用弃权来提高诚实性。在四个知识密集型基准上的大量实验表明，TruthRL显著减少了幻觉（例如，43.5% → 19.4%）并提高了诚实性（例如，5.3% → 37.2%），在各种骨干模型上均有一致的提升。分析表明，TruthRL的改进源于LLM识别其知识边界的能力增强，从而避免了像基线那样过于保守。
英文摘要：
While large language models (LLMs) have demonstrated strong performance on factoid question answering, they are still prone to hallucination and untruthful responses, particularly when tasks demand information outside their parametric knowledge. Indeed, truthfulness requires more than accuracy -- models must also recognize uncertainty and abstain when unsure to avoid hallucinations. This presents a fundamental challenge for existing methods: approaches that optimize for accuracy often amplify hallucinations, while those that encourage abstention can become overly conservative, sacrificing correct answers. Both extremes ultimately compromise truthfulness. In this work, we present TruthRL, a general reinforcement learning (RL) framework that directly optimizes the truthfulness of LLMs. Specifically, we implement TruthRL using GRPO with a simple yet effective ternary reward that distinguishes correct answers, hallucinations, and abstentions. It incentivizes models to reduce hallucinations not only by providing correct responses, but also by enabling abstention when uncertain, thereby improving truthfulness. Extensive experiments across four knowledge-intensive benchmarks show that TruthRL significantly reduces hallucinations (e.g., 43.5% $\rightarrow$ 19.4%) and improves truthfulness (e.g., 5.3% $\rightarrow$ 37.2%), with consistent gains across various backbone models. Analysis shows that the improvement of TruthRL arises from enhanced capability of LLMs to recognize their knowledge boundary, hence avoiding being overly conservative as the baselines are.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
