arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1902.11088cs.LGstat.ML
收藏

深部结构预测模型中的尺度问题

Scaling Matters in Deep Structured-Prediction Models

Samsung-HSE Laboratory(三星-高等经济学院实验室);National Research University Higher School of Economics(国立研究型大学高等经济学院)

Aleksandr Shevchenko, Anton Osokin

摘要:

基于能量的深度结构化预测模型结合了学习表示的表达能力和将有关当前任务的知识嵌入系统的能力。学习这种模型的参数的一种常见方法是多阶段过程,其中在不同阶段训练组件的不同组合。整个系统的端到端联合训练是最后的微调阶段。这种多级方法是耗时和繁琐的,因为它需要多次运行直到收敛和多轮超参数调整。从这一点来看,从一开始就启动联合训练程序是有益的。然而,这种方法往往出乎意料地失败,并提供比多级方法更差的结果。在本文中,我们假设基于深度能量的模型的联合训练失败的一个原因是能量函数中不同分量的不正确的相对归一化。我们提出了在线和离线缩放算法,这些算法修复了联合训练,并在三个不同的任务上证明了它们的有效性。

英文摘要:

Deep structured-prediction energy-based models combine the expressive power of learned representations and the ability of embedding knowledge about the task at hand into the system. A common way to learn parameters of such models consists in a multistage procedure where different combinations of components are trained at different stages. The joint end-to-end training of the whole system is then done as the last fine-tuning stage. This multistage approach is time-consuming and cumbersome as it requires multiple runs until convergence and multiple rounds of hyperparameter tuning. From this point of view, it is beneficial to start the joint training procedure from the beginning. However, such approaches often unexpectedly fail and deliver results worse than the multistage ones. In this paper, we hypothesize that one reason for joint training of deep energy-based models to fail is the incorrect relative normalization of different components in the energy function. We propose online and offline scaling algorithms that fix the joint training and demonstrate their efficacy on three different tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑