arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.11289cs.LGcs.AI
收藏

Recon:从根源上减少多任务学习的干扰

Recon: Reducing Conflicting Gradients from the Root for Multi-Task Learning

The Hong Kong Polytechnic University(香港理工大学)

Guangyuan Shi, Qimai Li, Wenlong Zhang, Jiaxin Chen, Xiao-Ming Wu

摘要:

多任务学习的一个基本挑战是,当不同的任务联合求解时,它们可能会相互冲突,而这种现象的原因是优化过程中的梯度冲突。最近的作品试图通过基于某些标准直接改变梯度来减轻冲突梯度的影响。然而,我们的实证研究表明,“梯度手术”不能有效地减少冲突梯度的发生。在本文中,我们采取了不同的方法来减少冲突的梯度从根。在本质上,我们调查的任务梯度w.r.t.每个共享网络层,选择具有高冲突分数的层,并将它们转换为特定于任务的层。我们的实验表明,这样一个简单的方法可以大大减少发生冲突的梯度在其余的共享层,并实现更好的性能,在许多情况下,只有轻微的增加模型参数。我们的方法可以很容易地应用于改进各种最先进的方法,包括梯度操作方法和分支结构搜索方法。给定网络体系结构(例如,ResNet18),它只需要搜索一次冲突层,并且可以修改网络,以便在相同甚至不同的数据集上使用不同的方法来提高性能。源代码可在https://github.com/moukamisama/Recon上获得。

英文摘要:

A fundamental challenge for multi-task learning is that different tasks may conflict with each other when they are solved jointly, and a cause of this phenomenon is conflicting gradients during optimization. Recent works attempt to mitigate the influence of conflicting gradients by directly altering the gradients based on some criteria. However, our empirical study shows that ``gradient surgery'' cannot effectively reduce the occurrence of conflicting gradients. In this paper, we take a different approach to reduce conflicting gradients from the root. In essence, we investigate the task gradients w.r.t. each shared network layer, select the layers with high conflict scores, and turn them to task-specific layers. Our experiments show that such a simple approach can greatly reduce the occurrence of conflicting gradients in the remaining shared layers and achieve better performance, with only a slight increase in model parameters in many cases. Our approach can be easily applied to improve various state-of-the-art methods including gradient manipulation methods and branched architecture search methods. Given a network architecture (e.g., ResNet18), it only needs to search for the conflict layers once, and the network can be modified to be used with different methods on the same or even different datasets to gain performance improvement. The source code is available at https://github.com/moukamisama/Recon.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑