arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.24844cs.LGcs.AIcs.CL
收藏

超越模式:语言模型中的分布推理强化学习

Reaching Beyond the Mode: RL for Distributional Reasoning in Language Models

Massachusetts Institute of Technology(麻省理工学院)

Isha Puri, Mehul Damani, Idan Shenfeld, Marzyeh Ghassemi, Jacob Andreas, Yoon Kim

AI总结:

本文提出一种多答案强化学习方法,使语言模型在推理时能生成多个可能的假设并评估其置信度,提升多样性和准确性。

摘要:

给定一个问题,语言模型(LM)隐含地编码了可能答案的分布。在实践中,训练后的过程常将此分布压缩为单一主导模式。尽管这在基准评估中通常不成问题,但许多现实任务本质上包含多个有效答案或不可约的不确定性。本文描述了一种多答案强化学习方法,用于训练语言模型在推理时进行多答案的分布推理。我们修改了强化学习目标,使模型能够在单次前向传递中显式生成多个候选答案,将推理时的搜索方面内化到模型的生成过程中。在问答、医疗诊断和编码基准中,我们观察到与单答案训练基线相比,多样性、覆盖度和集合级校准得分有所提高。使用本文方法训练的模型生成多个答案所需的token更少,且在编码任务中也更准确。这些结果表明,多答案强化学习是比推理时缩放程序如best-of-k的一种原理上和计算上更高效的选择。代码和更多信息可在https://multi-answer-rl.github.io/找到。

英文摘要:

Given a question, a language model (LM) implicitly encodes a distribution over possible answers. In practice, post-training procedures for LMs often collapse this distribution onto a single dominant mode. While this is generally not a problem for benchmark-style evaluations that assume one correct answer, many real-world tasks inherently involve multiple valid answers or irreducible uncertainty. Examples include medical diagnosis, ambiguous question answering, and settings with incomplete information. In these cases, we would like LMs to generate multiple plausible hypotheses, ideally with confidence estimates for each one, and without computationally intensive repeated sampling to generate non-modal answers. This paper describes a multi-answer reinforcement learning approach for training LMs to perform distributional reasoning over multiple answers during inference. We modify the RL objective to enable models to explicitly generate multiple candidate answers in a single forward pass, internalizing aspects of inference-time search into the model's generative process. Across question-answering, medical diagnostic, and coding benchmarks, we observe improved diversity, coverage, and set-level calibration scores compared to single answer trained baselines. Models trained with our approach require fewer tokens to generate multiple answers than competing approaches. On coding tasks, they are also substantially more accurate. These results position multi-answer RL as a principled and compute-efficient alternative to inference-time scaling procedures such as best-of-k. Code and more information can be found at https://multi-answer-rl.github.io/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑