arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.00441cs.LGcs.AIcs.CVcs.NE
收藏

基于局部任务相似性诱导神经元创建与移除的多任务结构学习

Multi-Task Structural Learning using Local Task Similarity induced Neuron Creation and Removal

NavInfo Europe(四维图新欧洲);Eindhoven University of Technology(埃因霍温理工大学)

Naresh Kumar Gurulingan, Bahram Zonooz, Elahe Arani

AI总结:

针对静态人工架构限制多任务学习泛化潜力的问题,提出MTSL方法交替进行任务学习与结构学习,通过局部相似任务层知识迁移至新建组层再移除原层的方式动态构建架构,实现了有竞争力的泛化效果与更强的分布外鲁棒性。

摘要:

多任务学习有望通过最大化任务间正向迁移同时减少任务干扰来提升泛化能力,但训练全程保持静态的人工设计架构阻碍了这一潜力的充分发挥。与之相反,大脑的学习过程伴随突触强度变化的同时,还会发生结构改变。因此,我们提出多任务结构学习(Multi-Task Structural Learning, MTSL)方法,可同时学习多任务架构及其参数。MTSL 为每个任务以相同的单任务网络启动,在任务学习阶段与结构学习阶段之间交替迭代。在任务学习阶段,每个网络针对对应任务实现专业化。在每个结构学习阶段,从最浅层开始,局部相似的任务层先将其知识迁移到一个新创建的组层,随后被移除;之后 MTSL 使用该组层替代对应的已移除任务层,再继续处理下一层。我们的实验结果表明,MTSL 在多种基线上取得了具有竞争力的泛化性能,并且提升了对分布外数据的鲁棒性。

英文摘要:

Multi-task learning has the potential to improve generalization by maximizing positive transfer between tasks while reducing task interference. Fully achieving this potential is hindered by manually designed architectures that remain static throughout training. On the contrary, learning in the brain occurs through structural changes that are in tandem with changes in synaptic strength. Thus, we propose \textit{Multi-Task Structural Learning (MTSL)} that simultaneously learns the multi-task architecture and its parameters. MTSL begins with an identical single-task network for each task and alternates between a task-learning phase and a structural-learning phase. In the task learning phase, each network specializes in the corresponding task. In each of the structural learning phases, starting from the earliest layer, locally similar task layers first transfer their knowledge to a newly created group layer before being removed. MTSL then uses the group layer in place of the corresponding removed task layers and moves on to the next layers. Our empirical results show that MTSL achieves competitive generalization with various baselines and improves robustness to out-of-distribution data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑