KnowRL: 探索知识增强型强化学习以提升事实性
KnowRL: Exploring Knowledgeable Reinforcement Learning for Factuality
AI总结：
KnowRL通过整合事实性奖励提升慢思考模型的事实准确性，实验表明其有效缓解了幻觉问题并保持推理能力。
摘要：
大型语言模型（LLMs），特别是慢思考模型，常表现出严重的幻觉，输出错误内容是因为无法准确识别知识边界。尽管强化学习（RL）可以增强复杂推理能力，但其结果导向的奖励机制缺乏对思考过程的事实监督，进一步加剧了幻觉问题。为解决慢思考模型的高幻觉问题，我们提出了知识增强型强化学习（KnowRL）。KnowRL通过在RL训练过程中整合基于知识验证的事实性奖励，引导模型进行基于事实的慢思考，帮助其识别知识边界。这种针对性的事实输入使模型能够学习和内化基于事实的推理策略。通过在推理步骤中直接奖励对事实的遵循，KnowRL促进了更可靠的思考过程。在三个幻觉评估数据集和两个推理评估数据集上的实验结果表明，KnowRL有效缓解了慢思考模型的幻觉问题，同时保持了其原有的强大推理能力。我们的代码可在https://github.com/zjunlp/KnowRL获取。
英文摘要：
Large Language Models (LLMs), particularly slow-thinking models, often exhibit severe hallucination, outputting incorrect content due to an inability to accurately recognize knowledge boundaries during reasoning. While Reinforcement Learning (RL) can enhance complex reasoning abilities, its outcome-oriented reward mechanism often lacks factual supervision over the thinking process, further exacerbating the hallucination problem. To address the high hallucination in slow-thinking models, we propose Knowledge-enhanced RL, KnowRL. KnowRL guides models to perform fact-based slow thinking by integrating a factuality reward, based on knowledge verification, into the RL training process, helping them recognize their knowledge boundaries. KnowRL guides models to perform fact-based slow thinking by integrating a factuality reward, based on knowledge verification, into the RL training process, helping them recognize their knowledge boundaries. This targeted factual input during RL training enables the model to learn and internalize fact-based reasoning strategies. By directly rewarding adherence to facts within the reasoning steps, KnowRL fosters a more reliable thinking process. Experimental results on three hallucination evaluation datasets and two reasoning evaluation datasets demonstrate that KnowRL effectively mitigates hallucinations in slow-thinking models while maintaining their original strong reasoning capabilities. Our code is available at https://github.com/zjunlp/KnowRL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
