arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.05077cs.CLcs.AI
收藏

Slm-mux: 通过协调小型语言模型进行推理

Slm-mux: Orchestrating small language models for reasoning

Harvard University(哈佛大学);Georgia Institute of Technology(佐治亚理工学院);Stanford University(斯坦福大学)

Chenyu Wang, Zishen Wan, Hao Kang, Emma Chen, Zhiqiang Xie, Tushar Krishna, Vijay Janapa Reddi, Yilun Du

AI总结:

SLM-MUX通过协调多个小型语言模型提升推理准确性,实现比单个模型更高的性能。

摘要:

随着语言模型的快速发展,小型语言模型(SLMs)的数量显著增加。尽管它们没有达到最先进的准确性,但它们更高效,且在特定任务上往往表现优异。这提出了一个自然的问题:能否将多个SLMs协调成一个系统,使每个模型都能有效贡献,从而实现比任何单个模型更高的准确性?现有的协调方法主要针对前沿模型(例如GPT-4),在应用于SLMs时表现欠佳。为了解决这一差距,我们提出了一种三种阶段的方法来协调SLMs。首先,我们引入SLM-MUX,一种多模型架构,能够有效协调多个SLMs。在此基础上,我们开发了两种优化策略:(i)一个模型选择搜索,从给定的池中识别最互补的SLMs;(ii)测试时的缩放,专门针对SLM-MUX。我们的方法取得了显著成果:与现有协调方法相比,我们的方法在MATH上提高了13.4%,在GPQA上提高了8.8%,在GSM8K上提高了7.0%。仅使用两个SLMs,SLM-MUX在GPQA和GSM8K上优于Qwen 2.5 72B,在MATH上表现相当。我们进一步提供了理论分析来证明我们方法的优势。额外的实验表明,SLM-MUX的核心原理可以扩展到开放性生成任务(例如HumanEval),并受益于其他模型类别,包括前沿LLMs和领域特定微调的SLMs。总之,我们证明了通过所提出的方法,SLMs可以被有效地协调成更准确和高效的系统。项目页面可在https://slm-mux.github.io/上找到。

英文摘要:

With the rapid development of language models, the number of small language models (SLMs) has grown significantly. Although they do not achieve state-of-the-art accuracy, they are more efficient and often excel at specific tasks. This raises a natural question: can multiple SLMs be orchestrated into a system where each contributes effectively, achieving higher accuracy than any individual model? Existing orchestration methods have primarily targeted frontier models (e.g., GPT-4) and perform suboptimally when applied to SLMs. To address this gap, we propose a three-stage approach for orchestrating SLMs. First, we introduce SLM-MUX, a multi-model architecture that effectively coordinates multiple SLMs. Building on this, we develop two optimization strategies: (i) a model selection search that identifies the most complementary SLMs from a given pool, and (ii) test-time scaling tailored to SLM-MUX. Our approach delivers strong results: Compared to existing orchestration methods, our approach achieves up to 13.4% improvement on MATH, 8.8% on GPQA, and 7.0% on GSM8K. With just two SLMs, SLM-MUX outperforms Qwen 2.5 72B on GPQA and GSM8K, and matches its performance on MATH. We further provide theoretical analyses to substantiate the advantages of our method. Additional experiments show that the core principle of SLM-MUX extends to open-ended generation tasks (e.g., HumanEval) and benefits other model classes, including frontier LLMs and domain-specific fine-tuned SLMs. In summary, we demonstrate that SLMs can be effectively orchestrated into more accurate and efficient systems through the proposed approach. The project page is available at https://slm-mux.github.io/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑