arXiv 2603.035352026-03-03cs.LG

参数高效专家中集成、融合与路由的权衡

Trade-offs in Ensembling, Merging and Routing Among Parameter-Efficient Experts

New York University（纽约大学）；Microsoft Research（微软研究院）

Sanae Lotfi, Lucas Caccia, Alessandro Sordoni, Jordan T. Ash, Miroslav Dudik

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

研究探讨了参数高效专家中集成、融合与路由的权衡，发现非均匀集成和融合提升性能，而路由提供更大收益，同时通过聚类和贪心子集选择降低计算成本。

摘要：

尽管通过轻量级适配器微调的大语言模型（LLMs）在多样任务上表现出色，但其在单个任务上的表现取决于微调策略。通过三种主要策略融合独立训练的不同强弱模型在多任务学习中显示出前景：集成，即结合独立模型的输出；融合，即通过参数平均融合模型权重；路由，即根据输入依赖性整合模型。然而，这些方法中的许多设计决策仍处于研究不足的状态，更复杂的集成、融合和路由技术的相对优势尚未完全理解。我们通过实验证讨它们的权衡，回答两个关键问题：超越均匀集成或融合有何优势？路由的灵活性是否值得其复杂性？我们的发现表明，非均匀集成和融合能提升性能，但路由提供更大的收益。为减轻路由的计算成本，我们分析专家选择技术，表明聚类和贪心子集选择可在最小开销下保持合理性能。这些见解推进了多任务学习模型融合的理解。

英文摘要：

While large language models (LLMs) fine-tuned with lightweight adapters achieve strong performance across diverse tasks, their performance on individual tasks depends on the fine-tuning strategy. Fusing independently trained models with different strengths has shown promise for multi-task learning through three main strategies: ensembling, which combines outputs from independent models; merging, which fuses model weights via parameter averaging; and routing, which integrates models in an input-dependent fashion. However, many design decisions in these approaches remain understudied, and the relative benefits of more sophisticated ensembling, merging and routing techniques are not fully understood. We empirically evaluate their trade-offs, addressing two key questions: What are the advantages of going beyond uniform ensembling or merging? And does the flexibility of routing justify its complexity? Our findings indicate that non-uniform ensembling and merging improve performance, but routing offers even greater gains. To mitigate the computational cost of routing, we analyze expert selection techniques, showing that clustering and greedy subset selection can maintain reasonable performance with minimal overhead. These insights advance our understanding of model fusion for multi-task learning.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文