arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.16587cs.LGcs.AIcs.HC
收藏

基于多功能奖励模型的高性价比在线多LLM选择

Cost-Effective Online Multi-LLM Selection with Versatile Reward Models

The Chinese University of Hong Kong(香港中文大学);Southeast University(东南大学);Carnegie Mellon University(卡内基梅隆大学);Huawei Technologies Co., Ltd.(华为技术有限公司)

Xiangxiang Dai, Jin Li, Xutong Liu, Anqi Yu, John C. S. Lui

AI总结:

本文提出C2MAB-V在线组合多臂老虎机框架,利用反馈、置信界与松弛舍入方法,在多LLM组合选择中平衡奖励、成本和约束,并给出理论保证与实验验证。

摘要:

随着大型语言模型(LLM)的快速发展,多LLM任务的多样性及其定价结构的可变性变得日益重要,因为不同LLM之间的成本可能差异巨大。为应对这些挑战,我们提出了C2MAB-V,这是一种具有多功能奖励模型的高性价比组合多臂老虎机(Cost-effective Combinatorial Multi-armed Bandit with Versatile reward models),用于最优的LLM选择与使用。该在线模型不同于传统的静态方法,也不同于那些依赖单一LLM且不考虑成本的方法。通过在调度云上部署多个LLM,并使用专门处理用户查询的本地服务器,C2MAB-V支持在组合搜索空间中选择多个LLM,专门适配具有不同奖励模型的各类协作任务类型。基于我们设计的在线反馈机制和置信界技术,C2MAB-V能够通过管理不同模型之间的探索-利用权衡,有效解决多LLM选择挑战,同时为多样化任务平衡成本与奖励。针对在权衡困境下选择多个LLM这一NP难整数线性规划问题,本文通过以下方式处理:i)由本地服务器将整数问题分解为松弛形式,ii)由调度云利用离散化舍入方案提供最优LLM组合,以及iii)基于反馈进行持续在线更新。理论上,我们证明C2MAB-V在多功能奖励模型上提供严格保证,并在某些退化情形下达到与最先进结果相匹配的遗憾和违约表现。实验上,我们展示了C2MAB-V在三个应用场景中使用九个LLM时,能够有效平衡性能与成本效率。

英文摘要:

With the rapid advancement of large language models (LLMs), the diversity of multi-LLM tasks and the variability in their pricing structures have become increasingly important, as costs can vary greatly between different LLMs. To tackle these challenges, we introduce the \textit{C2MAB-V}, a \underline{C}ost-effective \underline{C}ombinatorial \underline{M}ulti-armed \underline{B}andit with \underline{V}ersatile reward models for optimal LLM selection and usage. This online model differs from traditional static approaches or those reliant on a single LLM without cost consideration. With multiple LLMs deployed on a scheduling cloud and a local server dedicated to handling user queries, \textit{C2MAB-V} facilitates the selection of multiple LLMs over a combinatorial search space, specifically tailored for various collaborative task types with different reward models. Based on our designed online feedback mechanism and confidence bound technique, \textit{C2MAB-V} can effectively address the multi-LLM selection challenge by managing the exploration-exploitation trade-off across different models, while also balancing cost and reward for diverse tasks. The NP-hard integer linear programming problem for selecting multiple LLMs with trade-off dilemmas is addressed by: i) decomposing the integer problem into a relaxed form by the local server, ii) utilizing a discretization rounding scheme that provides optimal LLM combinations by the scheduling cloud, and iii) continual online updates based on feedback. Theoretically, we prove that \textit{C2MAB-V} offers strict guarantees over versatile reward models, matching state-of-the-art results for regret and violations in some degenerate cases. Empirically, we show that \textit{C2MAB-V} effectively balances performance and cost-efficiency with nine LLMs for three application scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑