通过拉格朗日重新参数化优化不可分解的数据相关正则化器提供了显着的性能和效率增益
Optimizing Nondecomposable Data Dependent Regularizers via Lagrangian Reparameterization offers Significant Performance and Efficiency Gains
摘要:
数据相关正则化被认为有益于机器学习中的各种问题。通常,这些正则化器不能容易地分解成有限数量项的和,例如,对单个示例项求和。$F_β$测量、ROC曲线下面积(AUCROC)和固定召回率下的精确度(P@R)是在许多应用中使用的一些突出示例。我们发现,对于大多数中大型数据集,可扩展性问题严重限制了我们利用这些正则化器的能力。重要的是,尽管最近取得了一些进展,但关键的技术障碍是,这些目标仍然难以通过反向传播程序进行优化。虽然一个有效的通用策略,这个问题仍然难以捉摸,在本文中,我们表明,对于许多数据相关的不可分解的正则化,在应用程序中,相当大的效率增益是可能的,最小的代码级别的变化;换句话说,没有专门的工具或数值计划是必要的。我们的程序涉及到一个重新参数化,然后是一个部分二元论-这导致了一个配方,具有可证明的廉价投影算子。我们提出了一个详细的分析,我们的算法的运行时间和收敛性能。在实验方面,我们表明,直接使用我们的计划显着提高了最先进的IOU措施报告的MSCOCO材料分割数据集的状态。
英文摘要:
Data dependent regularization is known to benefit a wide variety of problems in machine learning. Often, these regularizers cannot be easily decomposed into a sum over a finite number of terms, e.g., a sum over individual example-wise terms. The $F_β$ measure, Area under the ROC curve (AUCROC) and Precision at a fixed recall (P@R) are some prominent examples that are used in many applications. We find that for most medium to large sized datasets, scalability issues severely limit our ability in leveraging the benefits of such regularizers. Importantly, the key technical impediment despite some recent progress is that, such objectives remain difficult to optimize via backpropapagation procedures. While an efficient general-purpose strategy for this problem still remains elusive, in this paper, we show that for many data-dependent nondecomposable regularizers that are relevant in applications, sizable gains in efficiency are possible with minimal code-level changes; in other words, no specialized tools or numerical schemes are needed. Our procedure involves a reparameterization followed by a partial dualization -- this leads to a formulation that has provably cheap projection operators. We present a detailed analysis of runtime and convergence properties of our algorithm. On the experimental side, we show that a direct use of our scheme significantly improves the state of the art IOU measures reported for MSCOCO Stuff segmentation dataset.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。