arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.11686cs.CLcs.AI
收藏

CCoE:面向资源受限场景的多专家协作紧凑高效 LLM 框架

CCoE: A Compact and Efficient LLM Framework with Multi-Expert Collaboration for Resource-Limited Settings

Qihoo 360(奇虎360)

Shaomang Huang, Jianfeng Pan, Min Peng, Hanzhong Zheng

AI总结:

本文提出模块化多专家协作框架 CCoE,通过共享主干、独立专家子网络及规则化门控规划,在资源受限下实现五领域可比性能并降低内存与推理成本。

摘要:

大型语言模型(LLM)通过在海量数据集上训练,已在多个领域取得卓越性能。然而,扩展 LLM 以支持多个下游领域应用仍是一项重大挑战,尤其是在资源受限条件下。现有方法往往难以在多领域性能与资源效率之间取得平衡,从而限制了其更广泛的适用性。为解决这一问题,我们提出 CCoE 架构,这是一种可将领域专用专家无缝集成到统一 LLM 中的模块化框架。通过在共享主干分区上利用独立训练的专家子网络,CCoE 实现了最先进性能,同时显著降低多专家部署的资源需求。此外,CCoE 中基于规则的门控与专家规划能够实现灵活的任务分配,促进专家协作以处理复杂推理任务。CCoE 不仅降低推理成本,还为跨多种应用集成领域专业知识提供了灵活且可扩展的解决方案。在五个领域上的实验表明,CCoE 取得了与当前领域专用 LLM 相当的性能。此外,与现有多领域模型集成方法相比,CCoE 将内存使用降低 61.3%,同时相比参数高效的多专家集成方法,推理效率提升 0.76 倍。

英文摘要:

Large Language Models (LLMs) have achieved exceptional performance across diverse domains through training on massive datasets. However, scaling LLMs to support multiple downstream domain applications remains a significant challenge, especially under resource constraints. Existing approaches often struggle to balance performance across multiple domains with resource efficiency, limiting their broader applicability. To address this, we introduce the CCoE architecture, a modular framework that seamlessly integrates domain-specific experts into a unified LLM. By leveraging independently trained expert subnetworks on a shared backbone partition, CCoE achieves state-of-the-art performance while significantly reducing the resource requirements for multi-expert deployments. Furthermore, rule-based gating and expert planning in CCoE enable flexible task allocation, promoting expert collaboration to handle complex reasoning tasks. CCoE not only reduces inference costs but also provides a flexible and scalable solution for integrating domain expertise across diverse applications. Experiments on five domains demonstrate that CCoE achieves comparable performance to current domain-specific LLMs. Moreover, compared to existing multi-domain model ensemble methods, CCoE reduces memory usage by 61.3%, while improving inference efficiency by 0.76x over parameter-efficient multi-expert integration approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑