StepORLM：一种面向运筹学语言模型的带生成式过程监督的自进化框架
StepORLM: A Self-Evolving Framework With Generative Process Supervision For Operations Research Language Models
AI总结：
针对大语言模型求解运筹学问题时强化学习存在的信用分配与过程监督短视问题，提出带生成式过程监督的自进化框架StepORLM，通过策略模型与GenPRM协同进化及双反馈机制，在6个基准上取得SOTA，GenPRM还可提升多模型推理扩展性能。
摘要：
大语言模型（LLMs）在求解运筹学（OR）问题方面已展现出可观的能力。尽管强化学习是用于在OR问题上训练LLM的强大范式，但现有工作普遍存在两大关键局限。其一，结果奖励存在信用分配问题，正确的最终答案可能会强化有缺陷的推理过程。其二，传统的判别式过程监督具有短视性，无法从整体上评估OR建模中相互依赖的各个步骤。为此，我们提出了StepORLM，这是一种具备生成式过程监督的新型自进化框架。StepORLM的核心是一个协同进化循环，其中策略模型与生成式过程奖励模型（GenPRM）会迭代地相互优化。该循环由双反馈机制驱动：一是来自外部求解器的明确的、基于结果的验证，二是来自GenPRM的细致的、整体性的过程评估。组合后的信号通过加权直接偏好优化（W-DPO）用于对齐策略，同时也用于优化GenPRM。我们最终得到的参数量为8B的StepORLM在6个基准测试上创下了新的最优性能，显著优于规模大得多的通用模型、智能体方法以及专用基线模型。此外，协同进化得到的GenPRM可作为功能强大且适用范围广的过程验证器，大幅提升了我们的模型以及其他现有LLMs的推理扩展性能。
英文摘要：
Large Language Models (LLMs) have shown promising capabilities for solving Operations Research (OR) problems. While reinforcement learning serves as a powerful paradigm for LLM training on OR problems, existing works generally face two key limitations. First, outcome reward suffers from the credit assignment problem, where correct final answers can reinforce flawed reasoning. Second, conventional discriminative process supervision is myopic, failing to evaluate the interdependent steps of OR modeling holistically. To this end, we introduce StepORLM, a novel self-evolving framework with generative process supervision. At its core, StepORLM features a co-evolutionary loop where a policy model and a generative process reward model (GenPRM) iteratively improve on each other. This loop is driven by a dual-feedback mechanism: definitive, outcome-based verification from an external solver, and nuanced, holistic process evaluation from the GenPRM. The combined signal is used to align the policy via Weighted Direct Preference Optimization (W-DPO) and simultaneously refine the GenPRM. Our resulting 8B-parameter StepORLM establishes a new state-of-the-art across six benchmarks, significantly outperforming vastly larger generalist models, agentic methods, and specialized baselines. Moreover, the co-evolved GenPRM is able to act as a powerful and universally applicable process verifier, substantially boosting the inference scaling performance of both our own model and other existing LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
