Not all examples are created equal, but standard deep neural network training protocols treat each training point uniformly. Each example is propagated forward and backward through the network the same amount of times, independent of how much the example contributes to the learning protocol. Recent work has proposed ways to accelerate training by deviating from this uniform treatment. Popular methods entail up-weighting examples that contribute more to the loss with the intuition that examples with low loss have already been learned by the model, so their marginal value to the training procedure should be lower. This view assumes that updating the model with high loss examples will be beneficial to the model. However, this may not hold for noisy, real world data. In this paper, we theorize and then empirically demonstrate that loss-based acceleration methods degrade in scenarios with noisy and corrupted data. Our work suggests measures of example difficulty need to correctly separate out noise from other types of challenging examples.
Toward Efficient Testing of Graph Neural Networks via Test Input Prioritization
Computer Science and Technology(计算机科学与技术);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Integrated Circuits(集成电路);Zhejiang University(浙江大学);Software Engineering Institute(软件工程研究所);East China Normal University(华东师范大学)
Graph Attention Based Prioritization of Disease Responsible Genes from Multimodal Alzheimer's Network
Network Reconstruction & Analysis Lab and Department of Computer Applications, Sikkim University(网络重建与分析实验室和计算机应用系,西西米大学);Network Reconstruction & Analysis Lab, Department of Computer Applications, Sikkim University(网络重建与分析实验室,计算机应用系,西西米大学);Department of Computer Science & Engineering, Tezpur University(计算机科学与工程系,泰朱大学)
Deep Neural Network Based Accelerated Failure Time Models using Rank Loss
Department of Statistics (Institute of Applied Statistics)(统计学系(应用统计研究所));Jeonbuk National University(全北国立大学);Department of Applied Statistics(应用统计学系);Yonsei University(yonsei大学);Department of Statistics and Data Science(统计学与数据科学系);Center for Advanced Image and Information Technology(先进图像与信息技术中心)
Wasserstein GANs Work Because They Fail (to Approximate the Wasserstein Distance)
Cambridge Image Analysis Group, Department of Applied Mathematics and Theoretical Physics, University of Cambridge(剑桥大学应用数学与理论物理系剑桥图像分析组);Department of Mathematical Sciences, University of Bath(巴斯大学数学科学系);University of Cambridge(剑桥大学);University of Bath(巴斯大学)
Characterizing possible failure modes in physics-informed neural networks
Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室);University of California, Berkeley(加州大学伯克利分校);University of Utah(犹他大学);International Computer Science Institute(国际计算机科学研究所)