并非所有思考都同等重要：通过多轮强化学习实现高效的大语言模型推理
Not All Thoughts are Generated Equal: Efficient LLM Reasoning via Multi-Turn Reinforcement Learning
AI总结：
本文提出Long⊗Short框架，通过多轮强化学习让长思考与短思考LLM协作，区分思考重要性，在保持性能的同时减少超80%令牌长度。
摘要：
压缩大语言模型（LLM）中的长思维链（CoT）是提高LLM推理效率的一种新兴策略。尽管其前景广阔，但现有研究对长思维链中的所有思考均进行同等压缩，这阻碍了更简洁和有效的推理。为此，我们首先通过自动长思维链分块和蒙特卡洛展开，考察不同思考在推理中的有效性和效率，从而探究不同思考的重要性。基于这些见解，我们提出一个理论上有界的度量，以联合衡量不同思考的有效性和效率。随后，我们提出Long⊗Short，一个高效的推理框架，使两个LLM协作解决问题：一个长思考LLM用于更有效地生成重要思考，而一个短思考LLM用于高效生成其余思考。具体而言，我们首先合成少量冷启动数据，分别微调LLM以适应长思考和短思考的推理风格。此外，我们提出一种面向协同的多轮强化学习，专注于模型自我进化以及长思考与短思考LLM之间的协作。实验结果表明，我们的方法使Qwen2.5-7B和Llama3.1-8B在MATH500、AIME24/25、AMC23和GPQA Diamond基准上达到与DeepSeek-R1-Distill-Qwen-7B和DeepSeek-R1-Distill-Llama-8B相当的性能，同时将令牌长度减少超过80%。我们的数据和代码可在https://github.com/usail-hkust/LongShort获取。
英文摘要：
Compressing long chain-of-thought (CoT) from large language models (LLMs) is an emerging strategy to improve the reasoning efficiency of LLMs. Despite its promising benefits, existing studies equally compress all thoughts within a long CoT, hindering more concise and effective reasoning. To this end, we first investigate the importance of different thoughts by examining their effectiveness and efficiency in contributing to reasoning through automatic long CoT chunking and Monte Carlo rollouts. Building upon the insights, we propose a theoretically bounded metric to jointly measure the effectiveness and efficiency of different thoughts. We then propose Long$\otimes$Short, an efficient reasoning framework that enables two LLMs to collaboratively solve the problem: a long-thought LLM for more effectively generating important thoughts, while a short-thought LLM for efficiently generating remaining thoughts. Specifically, we begin by synthesizing a small amount of cold-start data to fine-tune LLMs for long-thought and short-thought reasoning styles, respectively. Furthermore, we propose a synergizing-oriented multi-turn reinforcement learning, focusing on the model self-evolution and collaboration between long-thought and short-thought LLMs. Experimental results show that our method enables Qwen2.5-7B and Llama3.1-8B to achieve comparable performance compared to DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B, while reducing token length by over 80% across the MATH500, AIME24/25, AMC23, and GPQA Diamond benchmarks. Our data and code are available at https://github.com/usail-hkust/LongShort.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
