arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.05748cs.LGcs.CL
收藏

GradTS:一种基于Transformer网络的自动辅助任务选择方法

GradTS: A Gradient-Based Automatic Auxiliary Task Selection Method Based on Transformer Networks

Dartmouth College(达特茅斯学院);Zhejiang University City College(浙江大学城市学院);Tsinghua University(清华大学)

Weicheng Ma, Renze Lou, Kai Zhang, Lili Wang, Soroush Vosoughi

摘要:

多任务学习研究的一个关键问题是如何自动选择高质量的辅助任务。提出了一种基于梯度计算的辅助任务自动选择方法GradTS。与AUTOSEM(一种强基线方法)相比,GradTS使用基于bert的后端模型将MT-DNN的性能从0.33%提高到17.93%,用于GLUE基准测试中的8个自然语言理解(NLU)任务。GradTS也是节省时间的,因为(1)它的梯度计算是基于单任务实验的,(2)当候选任务集改变时,梯度被重新使用,而无需额外的实验。例如,在8个GLUE分类任务上,GradTS的平均时间比AUTOSEM少21.32%,GPU消耗相当。此外,我们展示了GradTS在各种任务设置和模型选择中的鲁棒性,例如候选任务之间的混合目标。GradTS在这些案例研究中的效率和功效说明了其在MTL研究中的普遍适用性,而无需手动任务过滤或昂贵的参数调整。

英文摘要:

A key problem in multi-task learning (MTL) research is how to select high-quality auxiliary tasks automatically. This paper presents GradTS, an automatic auxiliary task selection method based on gradient calculation in Transformer-based models. Compared to AUTOSEM, a strong baseline method, GradTS improves the performance of MT-DNN with a bert-base-cased backend model, from 0.33% to 17.93% on 8 natural language understanding (NLU) tasks in the GLUE benchmarks. GradTS is also time-saving since (1) its gradient calculations are based on single-task experiments and (2) the gradients are re-used without additional experiments when the candidate task set changes. On the 8 GLUE classification tasks, for example, GradTS costs on average 21.32% less time than AUTOSEM with comparable GPU consumption. Further, we show the robustness of GradTS across various task settings and model selections, e.g. mixed objectives among candidate tasks. The efficiency and efficacy of GradTS in these case studies illustrate its general applicability in MTL research without requiring manual task filtering or costly parameter tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑