arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.09671cs.AIcs.CL
收藏

OpenR:一个用于大型语言模型高级推理的开源框架

OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models

University College London(伦敦大学学院);University of Liverpool(利物浦大学);Shanghai Jiao Tong University(上海交通大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Westlake University(西湖大学)

Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M. Ni, Linyi Yang, Ying Wen, Weinan Zhang

AI总结:

本文提出了开源框架OpenR,通过整合数据获取、强化学习训练、非自回归解码、测试时计算与过程监督,在MATH数据集上验证了其有效提升LLM推理能力的方法与贡献。

摘要:

在本技术报告中,我们介绍了OpenR,这是一个开源框架,旨在整合关键组件以增强大型语言模型(LLMs)的推理能力。OpenR将数据获取、强化学习训练(包括在线和离线)以及非自回归解码统一到一个内聚的软件平台中。我们的目标是建立一个开源平台和社区,以加速LLM推理的发展。受OpenAI的o1模型成功的启发,该模型通过逐步推理和强化学习展示了改进的推理能力,OpenR集成了测试时计算、强化学习和过程监督来提高LLM的推理。我们的工作是第一个提供开源框架的,该框架利用强化学习探索OpenAI o1模型的核心技术,实现了超越传统自回归方法的高级推理能力。我们通过在MATH数据集上评估OpenR,利用公开可用的数据和搜索方法,证明了OpenR的有效性。我们的初步实验证实了显著的收益,通过测试时计算和基于过程奖励模型的强化学习,推理和性能获得了相对改进。OpenR框架,包括代码、模型和数据集,可在https://openreasoner.github.io获取。

英文摘要:

In this technical report, we introduce OpenR, an open-source framework designed to integrate key components for enhancing the reasoning capabilities of large language models (LLMs). OpenR unifies data acquisition, reinforcement learning training (both online and offline), and non-autoregressive decoding into a cohesive software platform. Our goal is to establish an open-source platform and community to accelerate the development of LLM reasoning. Inspired by the success of OpenAI's o1 model, which demonstrated improved reasoning abilities through step-by-step reasoning and reinforcement learning, OpenR integrates test-time compute, reinforcement learning, and process supervision to improve reasoning in LLMs. Our work is the first to provide an open-source framework that explores the core techniques of OpenAI's o1 model with reinforcement learning, achieving advanced reasoning capabilities beyond traditional autoregressive methods. We demonstrate the efficacy of OpenR by evaluating it on the MATH dataset, utilising publicly available data and search methods. Our initial experiments confirm substantial gains, with relative improvements in reasoning and performance driven by test-time computation and reinforcement learning through process reward models. The OpenR framework, including code, models, and datasets, is accessible at https://openreasoner.github.io.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑