arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1904.11366cs.LGcs.AI
收藏

基于极限学习机的分散式多任务学习

Decentralized Multi-Task Learning Based on Extreme Learning Machines

Royal Institute of Technology (KTH)(皇家理工学院(KTH))

Yu Ye, Ming Xiao, Mikael Skoglund

摘要:

在多任务学习(MTL)中,相关任务联合学习以提高泛化性能。为了利用极端学习机(ELMs)的高学习速度,我们将ELM框架应用于MTL问题,其中所有任务的ELMs的输出权重是协作学习的。我们首先提出了ELM的MTL问题在集中式设置,这是解决建议MTL-ELM算法。由于不同任务的数据集是地理分布的,因此研究了分散式机器学习。我们制定了分散的MTL问题的基础上ELM优化多块耦合双凸目标函数。为了解决这个问题,我们提出了DMTL-ELM算法,这是一个混合的雅可比和高斯-赛德尔邻近多块交替方向乘子法(ADMM)。进一步,为了减少DMTL-ELM的计算量,提出了一阶近似的DMTL-ELM(FO-DMTL-ELM)。理论分析表明,DMTL-ELM和FO-DMTL-ELM的收敛性是有条件的。通过仿真,我们证明了建议MTL-ELM,DMTL-ELM,FO-DMTL-ELM算法的收敛性,也表明他们可以优于现有的MTL方法。此外,通过调整隐特征空间的维数,DMTL-ELM存在通信负载和学习精度之间的折衷。

英文摘要:

In multi-task learning (MTL), related tasks learn jointly to improve generalization performance. To exploit the high learning speed of extreme learning machines (ELMs), we apply the ELM framework to the MTL problem, where the output weights of ELMs for all the tasks are learned collaboratively. We first present the ELM based MTL problem in the centralized setting, which is solved by the proposed MTL-ELM algorithm. Due to the fact that many data sets of different tasks are geo-distributed, decentralized machine learning is studied. We formulate the decentralized MTL problem based on ELM as majorized multi-block optimization with coupled bi-convex objective functions. To solve the problem, we propose the DMTL-ELM algorithm, which is a hybrid Jacobian and Gauss-Seidel Proximal multi-block alternating direction method of multipliers (ADMM). Further, to reduce the computation load of DMTL-ELM, DMTL-ELM with first-order approximation (FO-DMTL-ELM) is presented. Theoretical analysis shows that the convergence to the stationary point of DMTL-ELM and FO-DMTL-ELM can be guaranteed conditionally. Through simulations, we demonstrate the convergence of proposed MTL-ELM, DMTL-ELM, and FO-DMTL-ELM algorithms, and also show that they can outperform existing MTL methods. Moreover, by adjusting the dimension of hidden feature space, there exists a trade-off between communication load and learning accuracy for DMTL-ELM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑