基于细粒度知识反馈的在线策略自对齐用于幻觉缓解
On-Policy Self-Alignment with Fine-grained Knowledge Feedback for Hallucination Mitigation
AI总结：
本文提出RLFH，一种在线策略自对齐方法，通过自我评估框架将回答分解为原子事实并生成细粒度奖励，以缓解大语言模型幻觉，实验验证其有效性。
摘要：
幻觉发生在大型语言模型在生成回答时表现出偏离其知识边界的行为。为解决这一关键问题，以往基于学习的方法尝试微调模型，但受限于离线策略采样和粗粒度反馈。在本文中，我们提出 RLFH（面向幻觉的强化学习），一种在线策略自对齐方法，使LLM能够主动探索其知识边界，并通过细粒度反馈信号自我修正生成行为。RLFH引入了一个自我评估框架，其中策略自身充当评判者。通过该框架，回答被自动分解为原子事实，并针对外部知识源评估其真实性和信息量。由此产生的语句级细粒度反馈随后被转化为词元级密集奖励信号。这使得在线强化学习能够在无需人工干预的情况下实现精确且及时的优化。在HotpotQA、SQuADv2和Biography基准上的全面评估验证了RLFH在幻觉缓解方面的有效性。
英文摘要：
Hallucination occurs when large language models exhibit behavior that deviates from the boundaries of their knowledge during response generation. To address this critical issue, previous learning-based methods attempt to finetune models but are limited by off-policy sampling and coarse-grained feedback. In this paper, we present \textit{\b{R}einforcement \b{L}earning \b{f}or \b{H}allucination} (RLFH), an on-policy self-alignment approach that enables LLMs to actively explore their knowledge boundaries and self-correct generation behavior through fine-grained feedback signals. RLFH introduces a self-assessment framework where the policy serves as its own judge. Through this framework, responses are automatically decomposed into atomic facts and their truthfulness and informativeness are assessed against external knowledge sources. The resulting fine-grained feedback at the statement level are then converted into token-level dense reward signals. This enables online reinforcement learning to achieve precise and timely optimization without human intervention. Comprehensive evaluations on HotpotQA, SQuADv2, and Biography benchmarks validate RLFH's effectiveness in hallucination mitigation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
