arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.19987cs.LGcs.AIcs.CL
收藏

突破大语言模型微调的性能上限:重新引入马尔可夫状态

Breaking the Capability Ceiling of LLM Post-Training by Reintroducing Markov States

UW-Madison(威斯康星大学麦迪逊分校)

Yurun Yuan, Tengyang Xie

AI总结:

本文提出通过引入马尔可夫状态突破大语言模型微调的性能上限,理论和实验均表明该方法能显著降低样本复杂度并提升复杂逻辑谜题的性能。

摘要:

强化学习(RL)已成为大语言模型(LLMs)微调和对齐的标准范式,但近期证据表明其面临持续的'能力上限':不同于经典RL系统发现新策略,RL用于LLMs往往只是对预训练权重中已存在的模式进行细化。在本工作中,我们识别出一个根本性的结构瓶颈:尽管经典RL依赖于紧凑且信息丰富的马尔可夫状态,当前LLM微调框架却受限于不断扩展的动作历史。我们重新审视一个长期存在于RL中的经典原则,但长期未被LLM微调所采用:显式马尔可夫状态。理论上,我们提供了严格的保证,证明利用估计的马尔可夫状态可以显著降低样本复杂度。实验上,我们表明引入马尔可夫状态能够一致突破标准RL微调的性能边界,跨一系列复杂逻辑谜题。我们的发现表明,超越'历史作为状态'的建模,转向结构化的马尔可夫表示,对于解锁生成AI中的开放性发现和真正的新推理能力至关重要。

英文摘要:

Reinforcement learning (RL) has become a standard paradigm for post-training and aligning Large Language Models (LLMs), yet recent evidence suggests it faces a persistent "capability ceiling": unlike classical RL systems that discover novel strategies, RL for LLMs often acts as a mere refiner of patterns already latent in pre-trained weights. In this work, we identify a fundamental structural bottleneck: while classical RL relies on compact, informative Markov states, current LLM post-training formulations are tethered to an ever-expanding history of actions. We revisit a classical principle long central to RL yet absent from LLM post-training: explicit Markov states. Theoretically, we provide rigorous guarantees demonstrating that leveraging estimated Markov states can significantly reduce sample complexity. Empirically, we show that introducing Markov states consistently breaks the performance boundaries of standard RL post-training across a suite of complex logic puzzles. Our findings suggest that moving beyond "history-as-state" modeling in favor of structured Markovian representations is essential for unlocking open-ended discovery and genuinely new reasoning capabilities in Generative AI.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑