SkillNet-NLU：一个用于通用自然语言理解的稀疏激活模型
SkillNet-NLU: A Sparsely Activated Model for General-Purpose Natural Language Understanding
摘要：
流行的深度模型是单一目的的，并且过度专注于单个任务。然而，当被扩展到新的任务时，他们通常会忘记以前学到的技能，从头开始学习。我们通过引入SkillNet-NLU来解决这个问题，SkillNet-NLU是一种通用模型，可以将现有技能缝合在一起，以更有效地学习新任务。我们方法的关键特征是，它是在预定义技能的指导下稀疏激活的。与传统的密集模型总是激活所有模型参数不同，SkillNet-NLU只激活与目标任务相关的部分模型参数。当学习新任务时，我们的方法精确地激活了所需的技能，并提供了添加新技能的选项。我们评估了自然语言理解任务，并有以下发现。首先，只有一个模型检查点，SkillNet-NLU的性能优于特定于任务的微调和两个多任务学习基线（即，密集模型和专家混合模型）。其次，稀疏激活的预训练进一步提高了整体性能。第三，SkillNet-NLU在扩展到新任务时，性能明显优于基线系统。
英文摘要：
Prevailing deep models are single-purpose and overspecialize at individual tasks. However, when being extended to new tasks, they typically forget previously learned skills and learn from scratch. We address this issue by introducing SkillNet-NLU, a general-purpose model that stitches together existing skills to learn new tasks more effectively. The key feature of our approach is that it is sparsely activated guided by predefined skills. Different from traditional dense models that always activate all the model parameters, SkillNet-NLU only activates parts of the model parameters whose skills are relevant to the target task. When learning for a new task, our approach precisely activates required skills and also provides an option to add new skills. We evaluate on natural language understandings tasks and have the following findings. First, with only one model checkpoint, SkillNet-NLU performs better than task-specific fine-tuning and two multi-task learning baselines (i.e., dense model and Mixture-of-Experts model) on six tasks. Second, sparsely activated pre-training further improves the overall performance. Third, SkillNet-NLU significantly outperforms baseline systems when being extended to new tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
