arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1410.6776cs.LGstat.ML
收藏

不可分解损失函数的在线随机梯度法

Online and Stochastic Gradient Methods for Non-decomposable Loss Functions

Microsoft Research(微软研究院);Indian Institute of Science(印度科学学院)

Purushottam Kar, Harikrishna Narasimhan, Prateek Jain

摘要:

在敏感领域的现代应用,如生物识别和医学经常需要使用不可分解的损失函数,如精度@k,F-测量等相比,点损失函数,如铰链损失,这些提供了更细粒度的控制预测,但在同一时间提出了新的挑战,在算法设计和分析。在这项工作中,我们发起了在线学习技术的研究,这样的不可分解的损失函数的目的是使增量学习,以及设计可扩展的批量问题求解器。为此,我们提出了一个在线学习框架,这样的损失函数。我们的模型享有几个很好的属性,其中主要是存在有效的在线学习算法与次线性遗憾和在线批量转换的界限。我们的模型是现有的点损失函数在线学习模型的可证明扩展。我们实例化两个流行的损失,prec@k和pAUC,在我们的模型中,并证明了他们两个的次线性遗憾界。我们的证明需要一个新的结构引理排名名单可能是独立的利益。然后,我们开发了可扩展的随机梯度下降求解不可分解的损失函数。我们证明了,对于满足一定的一致收敛性质(包括prec@k,pAUC和F-测度)的损失函数的大家庭,我们的方法可证明收敛到经验风险最小化。这种一致收敛的结果是不知道这些损失,我们建立这些使用新的证明技术。然后,我们在现实生活和基准数据集上进行了广泛的实验,以确定我们的方法可以比最近提出的切割平面方法快几个数量级。

英文摘要:

Modern applications in sensitive domains such as biometrics and medicine frequently require the use of non-decomposable loss functions such as precision@k, F-measure etc. Compared to point loss functions such as hinge-loss, these offer much more fine grained control over prediction, but at the same time present novel challenges in terms of algorithm design and analysis. In this work we initiate a study of online learning techniques for such non-decomposable loss functions with an aim to enable incremental learning as well as design scalable solvers for batch problems. To this end, we propose an online learning framework for such loss functions. Our model enjoys several nice properties, chief amongst them being the existence of efficient online learning algorithms with sublinear regret and online to batch conversion bounds. Our model is a provable extension of existing online learning models for point loss functions. We instantiate two popular losses, prec@k and pAUC, in our model and prove sublinear regret bounds for both of them. Our proofs require a novel structural lemma over ranked lists which may be of independent interest. We then develop scalable stochastic gradient descent solvers for non-decomposable loss functions. We show that for a large family of loss functions satisfying a certain uniform convergence property (that includes prec@k, pAUC, and F-measure), our methods provably converge to the empirical risk minimizer. Such uniform convergence results were not known for these losses and we establish these using novel proof techniques. We then use extensive experimentation on real life and benchmark datasets to establish that our method can be orders of magnitude faster than a recently proposed cutting plane method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑