BALTO: 用于幻觉缓解的平衡令牌级策略优化
BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation
AI总结：
针对大语言模型幻觉问题，提出BALTO框架，通过提取可验证事实声明并投影为令牌级标签，引入平衡信用分配机制，在六个模型-基准设置中实现最高忠实度，优于现有后训练基线。
摘要：
幻觉仍然是阻碍大语言模型在知识密集型环境中部署的主要障碍，在这些环境中，生成的响应必须忠实地基于所提供的证据。强化学习是缓解幻觉的一个有前景的方向，但响应级忠实度奖励存在粒度不匹配问题：局部幻觉可能导致受支持的内容受到虚假惩罚。尽管最近的工作引入了细粒度反馈，如声明级验证和令牌级奖励，但不平衡的信用分配仍可能引发长度、冗长或优化噪声偏差。我们提出了BALTO，一种用于幻觉缓解的平衡令牌级策略优化框架。BALTO提取可核查的事实声明，根据参考上下文对其进行验证，并将声明级判断投影到令牌级标签。该框架引入了一种平衡的令牌级信用分配机制。这种设计将概率质量从未受支持的内容重新分配到忠实内容，而不是抑制整个响应。我们从理论角度系统分析了响应级奖励的局限性，并证明了BALTO在幻觉缓解的训练稳定性和优化效率方面的优势。在ConFiQA、RAGTruth和FinLLM-Eval上的实验表明，BALTO在所有六个模型-基准设置中实现了最高的忠实度，并且在Q-Score上持续优于现有的后训练基线，展示了更强的忠实度-信息量权衡。
英文摘要：
Hallucinations remain a major obstacle to deploying large language models (LLMs) in knowledge-intensive settings, where generated responses must be faithfully grounded in provided evidence. Reinforcement learning (RL) is a promising direction for hallucination mitigation, but response-level faithfulness rewards suffer from a granularity mismatch: localized hallucinations can cause supported content to receive spurious penalties. Although recent work introduces fine-grained feedback such as claim-level verification and token-level rewards, unbalanced credit assignment can still induce length, verbosity, or optimization-noise biases. We propose BALTO, a Balanced Token-level Policy Optimization framework for hallucination mitigation. BALTO extracts checkable factual claims, verifies them against the reference context, and projects claim-level judgments to token-level labels. A balanced token-level credit assignment mechanism is introduced into the framework. This design redistributes probability mass from unsupported content toward faithful content, rather than suppressing the entire response. We systematically analyze the limitations of response-level rewards from a theoretical standpoint, and prove BALTO's advantages in training stability and optimization efficiency for hallucination mitigation. Experiments on ConFiQA, RAGTruth, and FinLLM-Eval show that BALTO achieves the highest faithfulness across all six model--benchmark settings and consistently outperforms existing post-training baselines in Q-Score, demonstrating a stronger faithfulness--informativeness trade-off.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
