Given a question, a language model (LM) implicitly encodes a distribution over possible answers. In practice, post-training procedures for LMs often collapse this distribution onto a single dominant mode. While this is generally not a problem for benchmark-style evaluations that assume one correct answer, many real-world tasks inherently involve multiple valid answers or irreducible uncertainty. Examples include medical diagnosis, ambiguous question answering, and settings with incomplete information. In these cases, we would like LMs to generate multiple plausible hypotheses, ideally with confidence estimates for each one, and without computationally intensive repeated sampling to generate non-modal answers. This paper describes a multi-answer reinforcement learning approach for training LMs to perform distributional reasoning over multiple answers during inference. We modify the RL objective to enable models to explicitly generate multiple candidate answers in a single forward pass, internalizing aspects of inference-time search into the model's generative process. Across question-answering, medical diagnostic, and coding benchmarks, we observe improved diversity, coverage, and set-level calibration scores compared to single answer trained baselines. Models trained with our approach require fewer tokens to generate multiple answers than competing approaches. On coding tasks, they are also substantially more accurate. These results position multi-answer RL as a principled and compute-efficient alternative to inference-time scaling procedures such as best-of-k. Code and more information can be found at https://multi-answer-rl.github.io/.
Distributional Biases in Post-Training: A Markovian Analysis of Reasoning Trajectories
City University of Hong Kong(香港城市大学);Center for Advanced Intelligence Project, RIKEN(RIKEN高级智能研究中心);The Institute of Statistical Mathematics(统计数学研究所);University of Sydney(悉尼大学);CFAR and IHPC, Agency for Science, Technology and Research (A*STAR)(A*STAR的CFAR和IHPC);Nanyang Technological University(南洋理工大学);The University of Tokyo(东京大学)
Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks
Machine Learning Research, Morgan Stanley, New York, NY(机器学习研究部,摩根士丹利,纽约,纽约);School of Electrical and Computer Engineering, Cornell University, Ithaca, NY(电气与计算机工程学院,康奈尔大学,伊萨卡,纽约)
Hidden States Know Where Reasoning Diverges: Credit Assignment via Span-Level Wasserstein Distance
Beijing University of Posts and Telecommunications(北京邮电大学);Meituan(美团);Tianjin University(天津大学);Renmin University of China(中国人民大学);The University of Melbourne(墨尔本大学);University of International Business and Economics(国际经贸大学)
Logic-Regularized Verifier Elicits Reasoning from LLMs
Department of Computer Science and Technology, East China Normal University(东华大学计算机科学与技术系);Institute of Artificial Intelligence (TeleAI), China Telecom(中国电信人工智能研究院(TeleAI))
InfoDensity: Rewarding Information-Dense Traces for Efficient Reasoning
Institute for Infocomm Research (I 2 R), A*STAR, Singapore(信息与通信研究机构(I 2 R),A*STAR,新加坡);Centre for Frontier AI Research (CFAR), A*STAR, Singapore(前沿人工智能研究中心(CFAR),A*STAR,新加坡)
PRTS: A Primitive Reasoning and Tasking System via Contrastive Representations
Institute of Artificial Intelligence (TeleAI), China Telecom(人工智能研究院(TeleAI),中国电信);Tsinghua University(清华大学);Shanghai Jiao Tong University(上海交通大学);Fudan University(复旦大学)
Stable Reasoning, Unstable Responses: Mitigating LLM Deception via Stability Asymmetry
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Beijing Academy of Artificial Intelligence(北京人工智能研究院);School of Chinese as a Second Language, Peking University(北京大学第二语言学院)
Reasoning in the Dark: Interleaved Vision-Text Reasoning in Latent Space
The Hong Kong Polytechnic University(香港理工大学);Singapore Management University(新加坡管理学院);Shandong University(山东大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))