Modern neural networks have been successful in many regression-based tasks such as face recognition, facial landmark detection, and image generation. In this work, we investigate an intuitive but understudied characteristic of modern neural networks, namely, the nonsmoothness. The experiments using synthetic data confirm that such operations as ReLU and max pooling in modern neural networks lead to nonsmoothness. We quantify the nonsmoothness using a feature named the sum of the magnitude of peaks (SMP) and model the input-output relationships for building blocks of modern neural networks. Experimental results confirm that our model can accurately predict the statistical behaviors of the nonsmoothness as it propagates through such building blocks as the convolutional layer, the ReLU activation, and the max pooling layer. We envision that the nonsmoothness feature can potentially be used as a forensic tool for regression-based applications of neural networks.
An explicit operator explains end-to-end computation in the modern neural networks used for sequence and language modeling
Department of Mathematics, Western University, London ON, Canada(温哥华大学数学系,伦敦ON,加拿大);Fields Lab for Network Computation, Fields Institute, Toronto ON, Canada(网络计算实验室,Fields研究所,多伦多ON,加拿大);King's University College at Western University, London ON, Canada(温哥华大学国王学院,伦敦ON,加拿大);Department of Neuroscience, University of Lethbridge, Lethbridge AB, Canada(利思堡大学神经科学系,利思堡AB,加拿大)
A Deep Latent Variable Framework for Jointly Modeling Missingness, Measurement Error, and Heterogeneity
University of Alberta(阿尔伯塔大学);University of Western Ontario(西安大略大学);Memorial Sloan Kettering Cancer Center(纪念斯隆凯特琳癌症中心);University of Prince Edward Island(爱德华王子岛大学)
Modern Neural Networks for Small Tabular Datasets: The New Default for Field-Scale Digital Soil Mapping?
Osnabrück University, Joint Lab Artificial Intelligence and Data Science(奥尔登堡大学,人工智能与数据科学联合实验室);Leibniz Institute for Agricultural Engineering and Bioeconomy (ATB)(莱比锡农业工程与生物经济研究所(ATB));German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心(DFKI))