PowerNet: Efficient Representations of Polynomials and Smooth Functions by Deep Neural Networks with Rectified Power Units
University of Chinese Academy of Sciences（中国科学院大学）；National Center for Mathematics and Interdisciplinary Sciences, Chinese Academy of Sciences（中国科学院国家数学与交叉科学中心）
Bo Li, Shanshan Tang, Haijun Yu
备注：
22 pages, 3 figures. arXiv admin note: text overlap with arXiv:1903.05858
Deep neural network with rectified linear units (ReLU) is getting more and more popular recently. However, the derivatives of the function represented by a ReLU network are not continuous, which limit the usage of ReLU network to situations only when smoothness is not required. In this paper, we construct deep neural networks with rectified power units (RePU), which can give better approximations for smooth functions. Optimal algorithms are proposed to explicitly build neural networks with sparsely connected RePUs, which we call PowerNets, to represent polynomials with no approximation error. For general smooth functions, we first project the function to their polynomial approximations, then use the proposed algorithms to construct corresponding PowerNets. Thus, the error of best polynomial approximation provides an upper bound of the best RePU network approximation error. For smooth functions in higher dimensional Sobolev spaces, we use fast spectral transforms for tensor-product grid and sparse grid discretization to get polynomial approximations. Our constructive algorithms show clearly a close connection between spectral methods and deep neural networks: a PowerNet with $n$ layers can exactly represent polynomials up to degree $s^n$, where $s$ is the power of RePUs. The proposed PowerNets have potential applications in the situations where high-accuracy is desired or smoothness is required.
ChebNet: Efficient and Stable Constructions of Deep Neural Networks with Rectified Power Units via Chebyshev Approximations
Software Development Center, Industrial and Commercial Bank of China（中国工商银行软件开发中心）；Hisilicon Semiconductor and Component Business Dept.(2012 Labs), Huawei Technologies Co., Ltd（华为技术有限公司半导体及组件业务部）；NCMIS & LSEC, Institute of Computational Mathematics and Scientific/Engineering Computing, Academy of Mathematics and Systems Science, Beijing（数学与系统科学研究院）；School of Mathematical Sciences, University of Chinese Academy of Sciences（中国科学院大学数学科学学院）
Why Rectified Power Unit Networks Fail and How to Improve It: An Effective Field Theory Perspective
School of Computational Sciences（计算科学学院）；Korea Institute for Advanced Study（韩国高级研究院）；Department of Mathematical Science（数学科学系）；Seoul National University（首尔国立大学）
Simons Center for Systems Biology（西蒙斯系统生物学中心）；Institute for Advanced Study（高等研究院）；Princeton Neuroscience Institute（普林斯顿神经科学研究所）；Princeton University（普林斯顿大学）
Top-N-Rank: A Scalable List-wise Ranking Method for Recommender Systems
Pennsylvania State University（宾夕法尼亚州立大学）；South China University of Technology（华南理工大学）
我们提出了Top-N-Rank，一种新颖的列表式学习排名模型，用于可靠地推荐N个排名靠前的项目。所提出的模型优化了广泛使用的折扣累积增益（DCG）目标函数的变体，其在两个重要方面不同于DCG：（i）它将DCG的评估仅限于排名列表中的前N个项目，从而消除了低排名项目对学习的排名函数的影响;以及（ii）它结合了允许模型利用具有不同可靠性或可信度级别的多种类型的隐式反馈的权重。由于得到的目标函数是非光滑的，因此优化具有挑战性，我们考虑使用传统的S形函数和校正线性单元（ReLU）对目标函数进行两个光滑近似。我们提出了一个家庭的学习排名算法（Top-N-Rank），工作与任何光滑的目标函数。然后，引入了一个更有效的变体Top-N-Rank.ReLU，该变体有效地利用了ReLU函数的性质，将Top-N-Rank的计算复杂性从用户评分的平均项目数的二次降低到线性。我们使用两个广泛使用的基准测试，即MovieLens数据集和Amazon Video Games数据集的实验结果表明：（i）目标函数的“top-N截断”大大提高了前N个推荐的排名质量;（ii）与使用sigmoid相比，使用ReLU来平滑目标函数在排名质量和运行时间方面都有显着改善;以及（iii）Top-N-Rank.ReLU在排名质量方面大大优于表现良好的列表式排名方法。
The Alan Turing Institute（图灵研究所）；Queen Mary University of London（伦敦玛丽女王大学）；Roche Pharmaceuticals（罗氏制药）；University of Southampton（南安普顿大学）；University College London Hospitals, NHS Foundation Trust（伦敦大学学院医院 NHS 基金会信托）
In neural networks , non-linearity is introduced by activation functions . One commonly used activation function is Rectified Linear Unit (ReLU). ReLU has been a popular choice as an activation but has flaws. State-of-the-art functions like Swish and Mish are now gaining attention as a better choice as they combat many flaws presented by other activation functions . CoLU is an activation function similar to Swish and Mish in properties. It is defined as f(x)=x/(1-xe^-(x+e^x)). It is smooth, continuously differentiable, unbounded above, bounded below, non-saturating, and non-monotonic. Based on…
