SLAW:用于高效多任务学习的标度损失近似加权
SLAW: Scaled Loss Approximate Weighting for Efficient Multi-Task Learning
摘要:
多任务学习(MTL)是机器学习的一个分支,具有重要的应用,但MTL中优化的多目标性质导致难以平衡任务之间的训练。最好的MTL优化方法需要单独计算每个任务的损失函数的梯度,这阻碍了大量任务的可扩展性。在本文中,我们提出了缩放损失近似加权(SLAW),多任务优化的方法,匹配的性能最好的现有方法,同时更有效。SLAW通过估计每个任务梯度的大小来平衡任务之间的学习,而不执行任何额外的向后传递。我们提供了理论和经验的理由SLAW的梯度幅度的估计。非线性回归、多任务计算机视觉和药物发现虚拟筛选的实验结果表明,SLAW在不牺牲性能的情况下比强基线显著更有效,并且适用于各种领域。
英文摘要:
Multi-task learning (MTL) is a subfield of machine learning with important applications, but the multi-objective nature of optimization in MTL leads to difficulties in balancing training between tasks. The best MTL optimization methods require individually computing the gradient of each task's loss function, which impedes scalability to a large number of tasks. In this paper, we propose Scaled Loss Approximate Weighting (SLAW), a method for multi-task optimization that matches the performance of the best existing methods while being much more efficient. SLAW balances learning between tasks by estimating the magnitudes of each task's gradient without performing any extra backward passes. We provide theoretical and empirical justification for SLAW's estimation of gradient magnitudes. Experimental results on non-linear regression, multi-task computer vision, and virtual screening for drug discovery demonstrate that SLAW is significantly more efficient than strong baselines without sacrificing performance and applicable to a diverse range of domains.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。