arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.26796cs.LGcs.AIstat.ML
收藏

在成本和容量限制下大型语言模型的鲁棒批量级查询路由

Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints

LinkedIn(领英);MIT(麻省理工学院)

Jelena Markovic-Voronov, Kayhan Behdin, Yuanda Xu, Zhengze Zhou, Zhipeng Wang, Rahul Mazumder

AI总结:

本文提出一种考虑资源的批量级查询路由框架,通过优化模型分配以提高鲁棒性和效率,实验显示在对抗性批处理下性能提升显著。

摘要:

我们研究了在成本、GPU资源和并发限制下将查询路由到大型语言模型(LLMs)的问题。传统单查询路由方法难以控制批量级成本,特别是在非均匀或对抗性批处理情况下。为此,我们提出了一种批量级、资源感知的路由框架,联合优化每个批次的模型分配,同时遵守成本和模型容量限制。我们进一步引入了考虑预测LLM性能不确定性的鲁棒变体,以及一种离线实例分配程序,平衡多个模型的质量和吞吐量。在两个多任务LLM基准测试中,鲁棒性在1-14%(取决于性能估计器)范围内提高了准确性,批量级路由在对抗性批处理下比单查询方法高出高达24%,而优化的实例分配相比非优化分配额外提高了3%。所有结果都在严格控制成本和GPU资源限制的情况下实现。

英文摘要:

We study the problem of routing queries to large language models (LLMs) under cost, GPU resources, and concurrency constraints. Prior per-query routing methods often fail to control batch-level cost, especially under non-uniform or adversarial batching. To address this, we propose a batch-level, resource-aware routing framework that jointly optimizes model assignment for each batch while respecting cost and model capacity limits. We further introduce a robust variant that accounts for uncertainty in predicted LLM performance, along with an offline instance allocation procedure that balances quality and throughput across multiple models. Experiments on two multi-task LLM benchmarks show that robustness improves accuracy by 1-14% over non-robust counterparts (depending on the performance estimator), batch-level routing outperforms per-query methods by up to 24% under adversarial batching, and optimized instance allocation yields additional gains of up to 3% compared to a non-optimized allocation, all while strictly controlling cost and GPU resource constraints.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑