LongCat-Flash-Prover:通过集成工具的强化学习推进原生形式推理
LongCat-Flash-Prover: Advancing Native Formal Reasoning via Agentic Tool-Integrated Reinforcement Learning
AI总结:
本文提出LongCat-Flash-Prover,一个5600亿参数的混合专家模型,通过集成工具的强化学习提升Lean4中的原生形式推理能力,实现了自动形式化、草图生成和证明三大能力,并通过混合专家迭代框架和分层重要性采样策略提升训练效率。
摘要:
我们介绍了LongCat-Flash-Prover,一个旗舰5600亿参数的混合专家(MoE)模型,通过集成工具的强化学习(TIR)推动Lean4中的原生形式推理。我们将原生形式推理任务分解为三个独立的形式能力,即自动形式化、草图生成和证明。为促进这些能力,我们提出混合专家迭代框架以扩展高质量任务轨迹,包括基于给定非正式问题生成形式化陈述、直接从陈述生成完整证明,或生成引理式草图。在代理强化学习中,我们提出分层重要性采样策略优化(HisPO)算法,旨在稳定MoE模型在长horizon任务上的训练。它采用梯度遮蔽策略,考虑策略陈旧性和内在训练-推理引擎差异,同时在序列和token层面。此外,我们还整合了定理一致性和合法性检测机制以消除奖励黑客问题。大量评估表明,我们的LongCat-Flash-Prover在自动形式化和定理证明方面为开放权重模型设定了新的状态。它在MiniF2F-Test上仅用72次推理预算即可达到97.1%的通过率。在更具有挑战性的基准测试中,它解决了ProverBench和PutnamBench的70.8%和41.5%,显著优于现有开放权重基线。
英文摘要:
We introduce LongCat-Flash-Prover, a flagship 560-billion-parameter open-source Mixture-of- Experts (MoE) model that advances Native Formal Reasoning in Lean4 through agentic tool-integrated reasoning (TIR). We decompose the native formal reasoning task into three independent formal capabilities, i.e., auto-formalization, sketching, and proving. To facilitate these capabilities, we propose a Hybrid-Experts Iteration Framework to expand high-quality task trajectories, including generating a formal statement based on a given informal problem, producing a whole-proof directly from the statement, or a lemma-style sketch. During agentic RL, we present a Hierarchical Importance Sampling Policy Optimization (HisPO) algorithm, which aims to stabilize the MoE model training on such long-horizon tasks. It employs a gradient masking strategy that accounts for the policy staleness and the inherent train-inference engine discrepancies at both sequence and token levels. Additionally, we also incorporate theorem consistency and legality detection mechanisms to eliminate reward hacking issues. Extensive evaluations show that our LongCat-Flash-Prover sets a new state-of-the-art for open-weights models in both auto-formalization and theorem proving. Demonstrating remarkable sample efficiency, it achieves a 97.1% pass rate on MiniF2F-Test using only 72 inference budget per problem. On more challenging benchmarks, it solves 70.8% of ProverBench and 41.5% of PutnamBench with no more than 220 attempts per problem, significantly outperforming existing open-weights baselines.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。