arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.04124cs.AIcond-mat.mtrl-scics.CLcs.LG
收藏

BeamPERL: 参数高效强化学习与可验证奖励用于结构梁力学推理

BeamPERL: Parameter-Efficient RL with Verifiable Rewards Specializes Compact LLMs for Structured Beam Mechanics Reasoning

Department of Mechanical Engineering(机械工程系);Massachusetts Institute of Technology(麻省理工学院);Department of Civil and Environmental Engineering(土木与环境工程系);Schwarzman College of Computing(施瓦茨曼计算学院)

Tarjei Paule Hage, Markus J. Buehler

AI总结:

BeamPERL通过参数高效强化学习与可验证奖励,提升紧凑语言模型在结构梁力学推理中的能力,发现精确奖励无法保证可转移的物理推理,需结合结构化推理支架以实现稳健科学推理。

摘要:

强化学习与可验证的硬奖励是否能教会一个紧凑的语言模型进行物理推理,还是主要学会向正确答案模式匹配?我们通过训练一个15亿参数的推理模型来研究这个问题,该模型在经典的工程问题——梁静力学上进行参数高效的RLVR训练,使用二进制正确性奖励来自符号求解器,而没有教师生成的推理轨迹。最好的BeamPERL检查点在Pass@1上比基模型提高了66.7%。然而,学习的能力是各向异性的:模型在组合上(更多负载)泛化,但在需要相同平衡方程的拓扑位移(移动支撑)下失败。中间检查点产生最强的推理能力,而继续优化会降低鲁棒性,同时保持奖励。这些发现揭示了结果层面对齐的关键限制:使用精确物理奖励的强化学习会诱导过程解决方案模板,而不是内部化支配方程。奖励信号的精度——即使分析上精确——本身并不保证可转移的物理推理。我们的结果表明,可验证的奖励可能需要与结构化的推理支架相结合,才能超越模板匹配,迈向稳健的科学推理。

英文摘要:

Can reinforcement learning with hard, verifiable rewards teach a compact language model to reason about physics, or does it primarily learn to pattern-match toward correct answers? We study this question by training a 1.5B-parameter reasoning model on beam statics, a classic engineering problem, using parameter-efficient RLVR with binary correctness rewards from symbolic solvers, without teacher-generated reasoning traces. The best BeamPERL checkpoint achieves a 66.7% improvement in Pass@1 over the base model. However, the learned competence is anisotropic: the model generalizes compositionally (more loads) but fails under topological shifts (moved supports) that require the same equilibrium equations. Intermediate checkpoints yield the strongest reasoning, while continued optimization degrades robustness while maintaining reward. These findings reveal a key limitation of outcome-level alignment: reinforcement learning with exact physics rewards induces procedural solution templates rather than internalization of governing equations. The precision of the reward signal - even when analytically exact - does not by itself guarantee transferable physical reasoning. Our results suggest that verifiable rewards may need to be paired with structured reasoning scaffolding to move beyond template matching toward robust scientific reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑