随着大型语言模型(LLM)的快速发展,多LLM任务的多样性及其定价结构的可变性变得日益重要,因为不同LLM之间的成本可能差异巨大。为应对这些挑战,我们提出了C2MAB-V,这是一种具有多功能奖励模型的高性价比组合多臂老虎机(Cost-effective Combinatorial Multi-armed Bandit with Versatile reward models),用于最优的LLM选择与使用。该在线模型不同于传统的静态方法,也不同于那些依赖单一LLM且不考虑成本的方法。通过在调度云上部署多个LLM,并使用专门处理用户查询的本地服务器,C2MAB-V支持在组合搜索空间中选择多个LLM,专门适配具有不同奖励模型的各类协作任务类型。基于我们设计的在线反馈机制和置信界技术,C2MAB-V能够通过管理不同模型之间的探索-利用权衡,有效解决多LLM选择挑战,同时为多样化任务平衡成本与奖励。针对在权衡困境下选择多个LLM这一NP难整数线性规划问题,本文通过以下方式处理:i)由本地服务器将整数问题分解为松弛形式,ii)由调度云利用离散化舍入方案提供最优LLM组合,以及iii)基于反馈进行持续在线更新。理论上,我们证明C2MAB-V在多功能奖励模型上提供严格保证,并在某些退化情形下达到与最先进结果相匹配的遗憾和违约表现。实验上,我们展示了C2MAB-V在三个应用场景中使用九个LLM时,能够有效平衡性能与成本效率。
英文摘要:
With the rapid advancement of large language models (LLMs), the diversity of multi-LLM tasks and the variability in their pricing structures have become increasingly important, as costs can vary greatly between different LLMs. To tackle these challenges, we introduce the \textit{C2MAB-V}, a \underline{C}ost-effective \underline{C}ombinatorial \underline{M}ulti-armed \underline{B}andit with \underline{V}ersatile reward models for optimal LLM selection and usage. This online model differs from traditional static approaches or those reliant on a single LLM without cost consideration. With multiple LLMs deployed on a scheduling cloud and a local server dedicated to handling user queries, \textit{C2MAB-V} facilitates the selection of multiple LLMs over a combinatorial search space, specifically tailored for various collaborative task types with different reward models. Based on our designed online feedback mechanism and confidence bound technique, \textit{C2MAB-V} can effectively address the multi-LLM selection challenge by managing the exploration-exploitation trade-off across different models, while also balancing cost and reward for diverse tasks. The NP-hard integer linear programming problem for selecting multiple LLMs with trade-off dilemmas is addressed by: i) decomposing the integer problem into a relaxed form by the local server, ii) utilizing a discretization rounding scheme that provides optimal LLM combinations by the scheduling cloud, and iii) continual online updates based on feedback. Theoretically, we prove that \textit{C2MAB-V} offers strict guarantees over versatile reward models, matching state-of-the-art results for regret and violations in some degenerate cases. Empirically, we show that \textit{C2MAB-V} effectively balances performance and cost-efficiency with nine LLMs for three application scenarios.
SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
Autonomous Learning, University of Tübingen(图宾根大学自主学习研究所);Empirical Inference, Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Neuro-Cognitive Modeling, University of Tübingen(图宾根大学神经认知建模研究所);VISLab, University of Amsterdam(阿姆斯特丹大学视觉实验室)
Task-Agnostic Pre-training and Task-Guided Fine-tuning for Versatile Diffusion Planner
Northwestern Polytechnical University(西北工业大学);Institute of Artificial Intelligence (TeleAI), China Telecom(人工智能研究院(TeleAI),中国电信);Tsinghua University(清华大学);Hong Kong University of Science(香港科学大学);Shenzhen Research Institute of Northwestern Polytechnical University(西北工业大学深圳研究院)
Online Subset Selection using $α$-Core with no Augmented Regret
Indian Institute of Technology Madras(印度理工学院马德拉斯分校);Chennai Mathematical Institute(金奈数学研究所);Indian Institute of Technology (ISM) Dhanbad(印度理工学院(印度矿业学院)丹巴德分校);Tata Institute of Fundamental Research(塔塔基础研究所)
Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Versatile Image Generation
Nanjing University(南京大学);Australia National University(澳大利亚国立大学);Wuhan University(武汉大学);National University of Singapore(新加坡国立大学);University of Technology Sydney(技术科技大学)
Minimal Ingredients for Reward Assignment from Expert Demonstrations
New York University(纽约大学);New York University Abu Dhabi(纽约大学阿布扎赫尔分校);Shanghai Frontiers Science Center of Artificial Intelligence and Deep Learning, NYU Shanghai(上海前沿科学中心(人工智能与深度学习))
Coordinating Ride-Pooling with Public Transit using Reward-Guided Conservative Q-Learning: An Offline Training and Online Fine-Tuning Reinforcement Learning Framework
The Hong Kong University of Science and Technology(香港科技大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Online Bipartite Matching with Reusable Capacity under Non-Stationary Rewards
Leonard N. Stern School of Business, New York University(纽约大学莱昂纳德·N.斯特恩商学院);Georgia Institute of Technology(佐治亚理工学院);Qiuzhen College, Tsinghua University(清华大学求真书院);Tsinghua University(清华大学);Yau Mathematical Sciences Center(丘成桐数学科学中心);Beijing Institute of Mathematical Sciences and Applications(北京数学科学与应用研究院)
Improving HPC Code Generation Capability of LLMs via Online Reinforcement Learning with Real-Machine Benchmark Rewards
Department of Computer Science, Nagoya University(名古屋大学计算机科学系);Graduate School of Informatics, Nagoya University(名古屋大学信息科学研究生院);Information Technology Center, Nagoya University(名古屋大学信息技术中心)