arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.00735cs.LGcs.AI
收藏

LADDER:通过递归问题分解实现LLM自我改进

LADDER: Self-Improving LLMs Through Recursive Problem Decomposition

Tufa Labs(图法实验室)

Toby Simonds, Akira Yoshiyama

AI总结:

提出LADDER框架,通过递归生成并解决复杂问题的简单变体实现LLM自我改进,并引入TTRL在推理时进行强化学习,在数学积分任务上大幅提升模型准确率并超越OpenAI o1。

摘要:

我们引入了LADDER(Learning through Autonomous Difficulty-Driven Example Recursion,通过自主难度驱动示例递归进行学习),这是一个使大型语言模型能够通过自我引导学习,递归地生成并解决复杂问题的更简单变体,从而自主提升其问题解决能力的框架。与以往需要精选数据集或人类反馈的方法不同,LADDER利用模型自身的能力来生成更简单的问题变体。我们在数学积分领域展示了LADDER的有效性,将Llama 3.2 3B在本科级问题上的准确率从1%提升至82%,并使Qwen2.5 7B Deepseek-R1 Distilled在MIT Integration Bee资格赛中达到73%。我们还引入了TTRL(Test-Time Reinforcement Learning,测试时强化学习),在推理时对测试问题的变体进行强化学习。TTRL使Qwen2.5 7B Deepseek-R1 Distilled在MIT Integration Bee资格赛中达到90%的最先进得分,超越了OpenAI o1的表现。这些结果表明,自主导向的策略学习如何在不依赖架构扩展或人类监督的情况下实现显著的能力提升。

英文摘要:

We introduce LADDER (Learning through Autonomous Difficulty-Driven Example Recursion), a framework which enables Large Language Models to autonomously improve their problem-solving capabilities through self-guided learning by recursively generating and solving progressively simpler variants of complex problems. Unlike prior approaches that require curated datasets or human feedback, LADDER leverages a model's own capabilities to generate easier question variants. We demonstrate LADDER's effectiveness in the subject of mathematical integration, improving Llama 3.2 3B's accuracy from 1% to 82% on undergraduate-level problems and enabling Qwen2.5 7B Deepseek-R1 Distilled to achieve 73% on the MIT Integration Bee qualifying examination. We also introduce TTRL (Test-Time Reinforcement Learning), where we perform reinforcement learning on variants of test problems at inference time. TTRL enables Qwen2.5 7B Deepseek-R1 Distilled to achieve a state-of-the-art score of 90% on the MIT Integration Bee qualifying examination, surpassing OpenAI o1's performance. These results show how self-directed strategic learning can achieve significant capability improvements without relying on architectural scaling or human supervision.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑