arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.18482cs.CLcs.AIcs.DBcs.IR
收藏

MixLLM:混合大型语言模型中的动态路由

MixLLM: Dynamic Routing in Mixed Large Language Models

Arizona State University(亚利桑那州立大学);NEC Labs America(美国NEC实验室)

Xinyuan Wang, Yanchi Liu, Wei Cheng, Xujiang Zhao, Zhengzhang Chen, Wenchao Yu, Yanjie Fu, Haifeng Chen

AI总结:

MixLLM提出基于动态上下文老虎机的路由系统,通过查询标签增强嵌入、轻量预测和元决策,实现质量、成本与延迟的最佳权衡,并支持持续学习。

摘要:

大型语言模型(LLMs)近期展现出人工通用智能的潜力,然而其使用成本高昂且响应延迟高。鉴于混合LLMs各自具有优缺点,LLM路由旨在为流中的每个查询识别最合适的模型,以最大化响应质量并最小化成本和延迟。然而,挑战包括:(1)质量、成本和延迟之间的动态权衡;(2)在部署系统中实现持续学习;(3)随时间导航变化的LLM候选集(例如,新增LLM或移除旧LLM)。为弥合这些差距,我们开发了MixLLM,一种基于动态上下文老虎机的路由系统,用于查询-LLM分配。具体来说,我们首先利用查询标签来增强查询嵌入以用于路由任务。接下来,我们设计轻量级预测模型来估计查询在LLM上的响应质量和成本。然后,我们设计一个元决策器来选择查询-LLM分配,以最佳地权衡响应质量、成本和延迟。最后,系统受益于持续训练,使其能够适应随时间变化的查询和用户反馈。我们的大量实验表明,MixLLM在响应质量、成本和延迟之间实现了最佳权衡(在时间约束下,达到GPT-4质量的97.25%,成本仅为24.18%)。

英文摘要:

Large Language Models (LLMs) exhibit potential artificial generic intelligence recently, however, their usage is costly with high response latency. Given mixed LLMs with their own strengths and weaknesses, LLM routing aims to identify the most suitable model for each query in the stream to maximize response quality and minimize cost and latency. However, the challenges involve: (1) dynamic trade-offs among quality, cost, and latency; (2) enabling continual learning in deployed systems; and (3) navigating a varying (e.g., new LLM addition or old LLM removal) set of LLM candidates over time. To bridge these gaps, we develop MixLLM, a dynamic contextual-bandit-based routing system for query-LLM assignment. Specifically, we first leverage query tags to enhance query embeddings for the routing task. Next, we design lightweight prediction models to estimate the response qualities and costs of queries over LLMs. We then devise a meta-decision maker to choose the query-LLM assignments to best tradeoff response quality, cost, and latency. Finally, the system benefits from continual training, allowing it to adapt to evolving queries and user feedback over time. Our extensive experiments show that MixLLM achieves the best trade-offs in response quality, cost, and latency (97.25% of GPT-4's quality at 24.18% of the cost under the time constraint).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑