arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.04700cs.CL
收藏

PRISM: 一种无需可验证奖励的统一后训练LLM框架

PRISM: A Unified Framework for Post-Training LLMs Without Verifiable Rewards

Arizona State University(亚利桑那州立大学);Amazon Web Services(亚马逊网络服务)

Mukesh Ghimire, Aosong Feng, Liwen You, Youzhi Luo, Fang Liu, Xuan Zhu

AI总结:

PRISM通过结合过程奖励模型与自我确定性,实现无需真实标签的稳定训练和提升LLM测试性能。

摘要:

当前后训练大型语言模型(LLM)的技术要么依赖于成本高昂的人类监督,要么依赖外部验证者来提升其在数学推理和代码生成等任务上的性能。然而,随着LLM问题解决能力的提升,任何进一步的改进都可能需要高质量的解决方案来解决困难问题,而这些解决方案人类无法获得。因此,在研究界,从无标签数据中学习变得越来越有吸引力。现有方法从模型的一致性中提取学习信号,要么通过多数投票,要么通过将模型的内部置信度转换为奖励。虽然内部一致性度量(如熵或自我确定性)不需要人类干预,但正如我们在本文中所展示的,这些在大规模和长期训练中是不可靠的信号。为了解决这种不可靠性,我们提出了PRISM,一种统一的训练框架,它使用过程奖励模型(PRM)来指导学习,同时在没有真实标签的情况下利用模型的内部置信度。我们表明,有效结合PRM与自我确定性可以实现稳定的训练和更好的测试性能,并且可以保持模型的内部置信度。代码可在https://github.com/ghimiremukesh/PRISM上获得。

英文摘要:

Current techniques for post-training Large Language Models (LLMs) rely either on costly human supervision or on external verifiers to boost performance on tasks such as mathematical reasoning and code generation. However, as LLMs improve their problem-solving, any further improvement will potentially require high-quality solutions to difficult problems that are not available to humans. As a result, learning from unlabeled data is becoming increasingly attractive in the research community. Existing methods extract learning signal from a model's consistency, either by majority voting or by converting the model's internal confidence into reward. Although internal consistency metric such as entropy or self-certainty require no human intervention, as we show in this work, these are unreliable signals for large-scale and long-term training. To address the unreliability, we propose PRISM, a unified training framework that uses a Process Reward Model (PRM) to guide learning alongside model's internal confidence in the absence of ground-truth labels. We show that effectively combining PRM with self-certainty can lead to both stable training and better test-time performance, and also keep the model's internal confidence in check. Code available at https://github.com/ghimiremukesh/PRISM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑