arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.04005cs.LG
收藏

多任务学习中梯度聚合的贝叶斯不确定性

Bayesian Uncertainty for Gradient Aggregation in Multi-Task Learning

Bar-Ilan University(巴伊兰大学);Sony Semiconductor Israel(索尼半导体以色列公司)

Idan Achituve, Idit Diamant, Arnon Netzer, Gal Chechik, Ethan Fetaya

AI总结:

针对多任务学习梯度聚合未考虑梯度维度敏感性的问题,提出基于贝叶斯推理的梯度聚合方法,通过任务参数分布量化梯度维度不确定性并用于聚合,在多数据集上达SOTA性能。

摘要:

随着机器学习的应用日益广泛,并行执行多个推理任务的需求不断增长。为每个任务运行专用模型的计算成本很高,因此多任务学习(MTL)受到了极大关注。MTL旨在学习一个能高效解决多个任务的单一模型。优化MTL模型通常通过为每个任务计算单个梯度并将其聚合以获得组合更新方向来实现。然而,这些方法未考虑一个重要方面:梯度维度的敏感性。本文提出了一种采用贝叶斯推理的新型梯度聚合方法。我们为任务特定参数设置概率分布,该分布进而生成任务梯度的分布。这一额外的宝贵信息使我们能够量化每个梯度维度的不确定性,进而在聚合梯度时将其纳入考量。我们在多个数据集上通过实验验证了该方法的优势,取得了当前最优性能。

英文摘要:

As machine learning becomes more prominent there is a growing demand to perform several inference tasks in parallel. Running a dedicated model for each task is computationally expensive and therefore there is a great interest in multi-task learning (MTL). MTL aims at learning a single model that solves several tasks efficiently. Optimizing MTL models is often achieved by computing a single gradient per task and aggregating them for obtaining a combined update direction. However, these approaches do not consider an important aspect, the sensitivity in the gradient dimensions. Here, we introduce a novel gradient aggregation approach using Bayesian inference. We place a probability distribution over the task-specific parameters, which in turn induce a distribution over the gradients of the tasks. This additional valuable information allows us to quantify the uncertainty in each of the gradients dimensions, which can then be factored in when aggregating them. We empirically demonstrate the benefits of our approach in a variety of datasets, achieving state-of-the-art performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑