提升推理语言模型的参数知识访问能力
Improving Parametric Knowledge Access in Reasoning Language Models
AI总结:
通过强化学习训练,提升推理语言模型在参数知识访问上的能力,改进知识回忆和问答任务表现。
摘要:
我们研究了语言模型在访问存储在参数中的世界知识时的推理能力。例如,回忆堪培拉是澳大利亚的首都可能受益于通过主要城市和专为首都设计的概念进行推理。尽管推理语言模型通过强化学习训练以在数学等任务上生成推理轨迹,但它们在访问自身世界知识时可能表现不佳。我们首先发现模型默认情况下不会生成最佳的世界知识推理:添加一个简单的“逐步思考”提示在知识回忆上显示出统计学上显著的改进,但对数学并不明显。受此启发,我们提出通过世界知识问答作为可验证的奖励来训练模型在参数知识上进行推理。在TriviaQA上训练后,性能在Natural Questions、HotpotQA、SimpleQA和StrategyQA上分别提高了4.2%、2.1%、0.6%和3.0%。推理模型在参数知识访问方面存在优化不足,但可以轻松地训练以更好地推理。
英文摘要:
We study reasoning for accessing world knowledge stored in a language model's parameters. For example, recalling that Canberra is Australia's capital may benefit from thinking through major cities and the concept of purpose-built capitals. While reasoning language models are trained via reinforcement learning to produce reasoning traces on tasks such as mathematics, they may not reason well for accessing their own world knowledge. We first find that models do not generate their best world knowledge reasoning by default: adding a simple "think step-by-step" cue demonstrates statistically significant improvement in knowledge recall but not math. Motivated by this, we propose training models to reason over their parametric knowledge using world-knowledge question answering as a verifiable reward. After reinforcement learning on TriviaQA (+9.9%), performance also improves on Natural Questions, HotpotQA, SimpleQA, and StrategyQA by 4.2%, 2.1%, 0.6%, and 3.0%, respectively. Reasoning models are under-optimized for parametric knowledge access, but can be easily trained to reason better.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。