通过智能训练加速极端分类的One-vs-All训练
Speeding-up One-vs-All Training for Extreme Classification via Smart Initialization
摘要:
在本文中,我们表明,一个简单的,数据依赖的方式设置的初始向量可以用来大大加快训练的线性一对所有(OVA)分类器在极端多标签分类(XMC)。从三个目标的角度讨论了初始权值的选取问题。我们希望在权重空间的一个区域中开始,a)具有低损失值,b)有利于二阶优化,c)可以快速执行共轭梯度(CG)计算。对于边际损失,这样的初始化是通过选择初始向量来实现的,这样它就可以将所有正(与标签相关)实例的平均值与所有负实例的平均值分开-这两个量可以快速计算出XMC中发生的高度不平衡的二进制问题。我们展示了$\approx 3\times$在各种XMC数据集上使用平方铰链损失进行训练的加速。这部分来自于由于开始更接近解决方案而需要执行的迭代次数减少,部分来自于允许在CG步骤中忽略容易的负面的隐式负面挖掘效果。由于优化问题的凸性质,在分类精度没有任何退化的情况下实现了加速。
英文摘要:
In this paper we show that a simple, data dependent way of setting the initial vector can be used to substantially speed up the training of linear one-versus-all (OVA) classifiers in extreme multi-label classification (XMC). We discuss the problem of choosing the initial weights from the perspective of three goals. We want to start in a region of weight space a) with low loss value, b) that is favourable for second-order optimization, and c) where the conjugate-gradient (CG) calculations can be performed quickly. For margin losses, such an initialization is achieved by selecting the initial vector such that it separates the mean of all positive (relevant for a label) instances from the mean of all negatives -- two quantities that can be calculated quickly for the highly imbalanced binary problems occurring in XMC. We demonstrate a speedup of $\approx 3\times$ for training with squared hinge loss on a variety of XMC datasets. This comes in part from the reduced number of iterations that need to be performed due to starting closer to the solution, and in part from an implicit negative mining effect that allows to ignore easy negatives in the CG step. Because of the convex nature of the optimization problem, the speedup is achieved without any degradation in classification accuracy.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。