arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.11877cs.CLcs.AI
收藏

迈向协作大语言模型系统中路由器的公平且全面评估

Towards Fair and Comprehensive Evaluation of Routers in Collaborative LLM Systems

Southern University of Science and Technology(南方科技大学);Institut Polytechnique de Paris(巴黎政治学院);Peking University(北京大学);Deepexi Technology Co. Ltd.(深醒科技有限公司);University of Edinburgh(爱丁堡大学);Beihang University(北航);Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳))

Wanxing Wu, He Zhu, Yixia Li, Lei Yang, Jiehui Zhao, Hongru Wang, Jian Yang, Benyou Wang, Bingyi Jing, Guanhua Chen

AI总结:

本文提出RouterXBench框架和ProbeDirichlet路由器,通过内部隐藏状态提升路由能力与跨领域鲁棒性,实现对协作大语言模型系统中路由器的公平全面评估。

摘要:

大型语言模型(LLMs)已取得成功,但成本和隐私约束要求在本地部署较小的模型,同时将复杂的查询卸载到基于云的模型上。现有的路由器评估是无系统的,忽略了场景特定的需求和分布外的鲁棒性。我们提出了RouterXBench,一个有原则的评估框架,包含三个维度:路由能力、场景对齐性和跨领域鲁棒性。与以往依赖输出概率或外部嵌入的工作不同,我们利用内部隐藏状态,这些状态在回答生成之前捕捉了模型的不确定性。我们引入了ProbeDirichlet,一种轻量级的路由器,通过可学习的狄利克雷分布聚合跨层隐藏状态,具有概率性训练。在多领域数据上训练,它在域内和分布外场景中表现出鲁棒的泛化能力。我们的结果表明,ProbeDirichlet在路由能力和高准确度场景中分别比最佳基线提高了16.68%和18.86%,在模型家族、模型规模、异构任务和代理工作流中表现出一致的性能。

英文摘要:

Large language models (LLMs) have achieved success, but cost and privacy constraints necessitate deploying smaller models locally while offloading complex queries to cloud-based models. Existing router evaluations are unsystematic, overlooking scenario-specific requirements and out-of-distribution robustness. We propose RouterXBench, a principled evaluation framework with three dimensions: router ability, scenario alignment, and cross-domain robustness. Unlike prior work that relies on output probabilities or external embeddings, we utilize internal hidden states that capture model uncertainty before answer generation. We introduce ProbeDirichlet, a lightweight router that aggregates cross-layer hidden states via learnable Dirichlet distributions with probabilistic training. Trained on multi-domain data, it generalizes robustly across in-domain and out-of-distribution scenarios. Our results show ProbeDirichlet achieves 16.68% and 18.86% relative improvements over the best baselines in router ability and high-accuracy scenarios, with consistent performance across model families, model scales, heterogeneous tasks, and agentic workflows.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑