arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2204.07689cs.LGcs.CL
收藏

稀疏激活的混合专家模型是鲁棒的多任务学习器

Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners

Microsoft(微软);Google(谷歌)

Shashank Gupta, Subhabrata Mukherjee, Krishan Subudhi, Eduardo Gonzalez, Damien Jose, Ahmed H. Awadallah, Jianfeng Gao

摘要:

传统的多任务学习(MTL)方法使用密集网络,这些网络在几个不同的任务中使用相同的共享权重集。这通常会在两个或多个任务竞争在不同方向上提取模型参数时产生干扰。在这项工作中,我们研究是否稀疏激活的混合专家模型(MoE)提高多任务学习,专门学习共享表示的一些权重,并使用其他学习任务特定的信息。为此,我们设计了任务感知的门控函数,将来自不同任务的示例路由到专门的专家,这些专家共享基于任务的网络权重子集。这导致了具有大量参数的稀疏激活多任务模型,但具有与密集模型相同的计算成本。我们展示了这样的稀疏网络,以改善多任务学习沿三个关键方面:(i)转移到低资源的任务,从相关的任务在训练混合;(ii)样本有效的泛化到任务在训练过程中没有看到利用任务感知路由从看到相关的任务;(iii)鲁棒性添加无关的任务,避免灾难性遗忘现有的任务。

英文摘要:

Traditional multi-task learning (MTL) methods use dense networks that use the same set of shared weights across several different tasks. This often creates interference where two or more tasks compete to pull model parameters in different directions. In this work, we study whether sparsely activated Mixture-of-Experts (MoE) improve multi-task learning by specializing some weights for learning shared representations and using the others for learning task-specific information. To this end, we devise task-aware gating functions to route examples from different tasks to specialized experts which share subsets of network weights conditioned on the task. This results in a sparsely activated multi-task model with a large number of parameters, but with the same computational cost as that of a dense model. We demonstrate such sparse networks to improve multi-task learning along three key dimensions: (i) transfer to low-resource tasks from related tasks in the training mixture; (ii) sample-efficient generalization to tasks not seen during training by making use of task-aware routing from seen related tasks; (iii) robustness to the addition of unrelated tasks by avoiding catastrophic forgetting of existing tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑