arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.05903cs.CL
收藏

HAPS: 基于联合架构和参数搜索的分层LLM路由

HAPS: Hierarchical LLM Routing with Joint Architecture and Parameter Search

Renmin University of China(中国人民大学);Wireless Technology Lab, Huawei Technologies Co., Ltd.(华为技术有限公司无线技术实验室)

Zihang Tian, Rui Li, Jingsen Zhang, Xiaohe Bo, Wei Huo, Xu Chen

AI总结:

HAPS通过联合搜索模型架构和参数,提升大型语言模型在多样化任务中的路由性能。

摘要:

大型语言模型(LLM)路由旨在利用不同LLM在多样化任务中的专业优势。然而,现有方法通常专注于选择LLM架构而忽视参数设置,这些参数对于任务性能至关重要。在本文中,我们引入了HAPS,一种分层LLM路由框架,该框架同时搜索模型架构和参数。具体来说,我们使用高层路由器在候选LLM架构中进行选择,然后根据低层路由器搜索所选架构的最佳参数。我们设计了一个参数生成网络,使两个路由器之间共享参数,以相互增强其能力。在训练过程中,我们设计了一个奖励增强的目标,以有效优化我们的框架。在两个常用基准测试上的实验表明,HAPS在强路由基线中表现一致更优。我们已发布代码至https://github.com/zihangtian/HAPS。

英文摘要:

Large language model (LLM) routing aims to exploit the specialized strengths of different LLMs for diverse tasks. However, existing approaches typically focus only on selecting LLM architectures, treating candidate models as fixed while overlooking parameter adaptation, which can substantially affect task performance. In this paper, we introduce HAPS, a hierarchical LLM routing framework that jointly searches over model architectures and parameters. Specifically, a high-level router selects among candidate LLM architectures, while a low-level router generates input-conditioned LoRA parameters for the selected architecture, expanding each candidate from a single fixed behavior into a region of reachable behaviors. The low-level router is built on the backbone of the high-level router, so that architecture selection and parameter generation share a common representation of the prompt. We further optimize the whole framework with a reinforcement learning objective. Extensive experiments show that HAPS consistently outperforms strong routing baselines across reasoning- and knowledge-intensive benchmarks, with analyses of its components, design choices, scalability, cost, and mixed open- and closed-source deployment further demonstrating the robustness of these gains. Our code is available at https://github.com/zihangtian/HAPS.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑