学习进行事实性推理
Learning to Reason for Factuality
AI总结：
研究推理大型语言模型在事实性推理方面的问题，提出同时考虑事实精度、响应细节和答案相关性的新型奖励函数，应用在线强化学习，使模型在长篇事实性基准测试中幻觉率降低、答案细节提升且响应帮助性无降。
摘要：
推理大型语言模型（R-LLMs）显著推进了复杂推理任务，但在事实性方面常常存在问题，在长篇事实性基准测试中产生的幻觉比非推理模型多得多。将在线强化学习（RL）扩展到长篇事实性设置面临挑战，此前工作利用FActScore等自动事实性评估框架在离线RL设置中策划偏好数据，但直接将此类方法用作在线RL奖励会导致奖励作弊。本文提出一种同时考虑事实精度、响应细节水平和答案相关性的新型奖励函数，并应用在线RL学习高质量事实性推理。在六个长篇事实性基准测试上评估，我们的事实性推理模型幻觉率平均降低23.1个百分点，答案细节水平提高23%，整体响应帮助性无下降。
英文摘要：
Reasoning Large Language Models (R-LLMs) have significantly advanced complex reasoning tasks but often struggle with factuality, generating substantially more hallucinations than their non-reasoning counterparts on long-form factuality benchmarks. However, extending online Reinforcement Learning (RL), a key component in recent R-LLM advancements, to the long-form factuality setting poses several unique challenges due to the lack of reliable verification methods. Previous work has utilized automatic factuality evaluation frameworks such as FActScore to curate preference data in the offline RL setting, yet we find that directly leveraging such methods as the reward in online RL leads to reward hacking in multiple ways, such as producing less detailed or relevant responses. We propose a novel reward function that simultaneously considers the factual precision, response detail level, and answer relevance, and applies online RL to learn high quality factual reasoning. Evaluated on six long-form factuality benchmarks, our factual reasoning model achieves an average reduction of 23.1 percentage points in hallucination rate, a 23% increase in answer detail level, and no degradation in the overall response helpfulness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
