SkillNet-X：一种具有稀疏激活技能的多语言多任务模型
SkillNet-X: A Multilingual Multitask Model with Sparsely Activated Skills
AI总结：
本文提出多语言多任务模型 SkillNet-X，通过稀疏激活语言和任务特定的技能模块作为知识中转枢纽，在多语言多任务场景下优于基线模型并具备良好泛化能力。
摘要：
传统的多任务学习方法基本上只能在任务或语言层面利用共性知识，这会丢失跨语言或跨任务知识。本文提出了一种通用的多语言多任务模型，名为 SkillNet-X，它使单一模型能够处理来自不同语言的许多不同任务。为此，我们定义了若干语言特定技能和任务特定技能，每个技能对应一个技能模块。SkillNet-X 稀疏地激活与目标任务或目标语言相关的部分技能模块。作为知识中转枢纽，技能模块能够连续吸收任务相关知识和语言相关知识。基于 Transformer，我们修改了多头注意力层和前馈网络层以适应技能模块。我们在四种语言的十一个自然语言理解数据集上评估了 SkillNet-X。结果表明，SkillNet-X 的表现优于任务特定基线和两个多任务学习基线（即稠密联合模型和 Mixture-of-Experts 模型）。此外，技能预训练进一步提升了 SkillNet-X 在几乎所有数据集上的性能。为了研究模型的泛化能力，我们在两个新任务上进行了实验，发现 SkillNet-X 显著优于基线。
英文摘要：
Traditional multitask learning methods basically can only exploit common knowledge in task- or language-wise, which lose either cross-language or cross-task knowledge. This paper proposes a general multilingual multitask model, named SkillNet-X, which enables a single model to tackle many different tasks from different languages. To this end, we define several language-specific skills and task-specific skills, each of which corresponds to a skill module. SkillNet-X sparsely activates parts of the skill modules which are relevant either to the target task or the target language. Acting as knowledge transit hubs, skill modules are capable of absorbing task-related knowledge and language-related knowledge consecutively. Based on Transformer, we modify the multi-head attention layer and the feed forward network layer to accommodate skill modules. We evaluate SkillNet-X on eleven natural language understanding datasets in four languages. Results show that SkillNet-X performs better than task-specific baselines and two multitask learning baselines (i.e., dense joint model and Mixture-of-Experts model). Furthermore, skill pre-training further improves the performance of SkillNet-X on almost all datasets. To investigate the generalization of our model, we conduct experiments on two new tasks and find that SkillNet-X significantly outperforms baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
