arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1903.01182cs.LGcs.CVstat.ML
收藏

补充目标培训

Complement Objective Training

Department of Computer Science, National Tsing-Hua University(国立清华大学计算机科学系);Electronic and Optoelectronic System Research Laboratories, ITRI(工研院电子与光电系统研究实验室);Google Research, Mountain View, CA, USA(谷歌研究院(美国加州山景城))

Hao-Yun Chen, Pei-Hsin Wang, Chun-Hao Liu, Shih-Chieh Chang, Jia-Yu Pan, Yu-Ting Chen, Wei Wei, Da-Cheng Juan

摘要:

具有主要目标的学习,例如用于分类和序列生成的softmax交叉熵,多年来一直是训练深度神经网络的规范。虽然是一种被广泛采用的方法,但使用交叉熵作为主要目标主要利用来自地面实况类的信息来最大化数据可能性,并且在很大程度上忽略了来自补充(不正确)类的信息。我们认为,除了主要目标之外,还使用补充目标进行训练,该补充目标利用来自补充类的信息,可以有效地提高模型性能。这促使我们研究一种新的训练范式,该范式最大化地面实况类的可能性,同时中和补充类的概率。我们对从计算机视觉到自然语言理解的多个任务进行了广泛的实验。实验结果证实,与仅具有一个主要目标的传统训练相比,具有补充目标的训练进一步提高了最先进模型在所有任务中的性能。除了准确性的提高,我们还表明,用主要目标和补充目标训练的模型对单步对抗攻击更鲁棒。

英文摘要:

Learning with a primary objective, such as softmax cross entropy for classification and sequence generation, has been the norm for training deep neural networks for years. Although being a widely-adopted approach, using cross entropy as the primary objective exploits mostly the information from the ground-truth class for maximizing data likelihood, and largely ignores information from the complement (incorrect) classes. We argue that, in addition to the primary objective, training also using a complement objective that leverages information from the complement classes can be effective in improving model performance. This motivates us to study a new training paradigm that maximizes the likelihood of the groundtruth class while neutralizing the probabilities of the complement classes. We conduct extensive experiments on multiple tasks ranging from computer vision to natural language understanding. The experimental results confirm that, compared to the conventional training with just one primary objective, training also with the complement objective further improves the performance of the state-of-the-art models across all tasks. In addition to the accuracy improvement, we also show that models trained with both primary and complement objectives are more robust to single-step adversarial attacks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑