A deep neural network using rectified linear units represents a continuous piecewise linear (CPWL) function and vice versa. Recent results in the literature estimated that the number of neurons needed to exactly represent any CPWL function grows exponentially with the number of pieces or exponentially in terms of the factorial of the number of distinct linear components. Moreover, such growth is amplified linearly with the input dimension. These existing results seem to indicate that the cost of representing a CPWL function is expensive. In this paper, we propose much tighter bounds and establish a polynomial time algorithm to find a network satisfying these bounds for any given CPWL function. We prove that the number of hidden neurons required to exactly represent any CPWL function is at most a quadratic function of the number of pieces. In contrast to all previous results, this upper bound is invariant to the input dimension. Besides the number of pieces, we also study the number of distinct linear components in CPWL functions. When such a number is also given, we prove that the quadratic complexity turns into bilinear, which implies a lower neural complexity because the number of distinct linear components is always not greater than the minimum number of pieces in a CPWL function. When the number of pieces is unknown, we prove that, in terms of the number of distinct linear components, the neural complexities of any CPWL function are at most polynomial growth for low-dimensional inputs and factorial growth for the worst-case scenario, which are significantly better than existing results in the literature.
Representing Piecewise-Linear Functions by Functions with Minimal Arity
Johann Radon Institute for Computational and Applied Mathematics(约翰·拉登研究所(计算与应用数学));Research Institute for Symbolic Computation(符号计算研究所);Johannes Kepler University(约翰· Kepler大学)
Department of Theoretical Neuroscience, Central Institute of Mental Health, Mannheim, Germany(理论神经科学系,心理健康中央研究所,曼海姆,德国);Faculty of Physics and Astronomy, Heidelberg University, Heidelberg, Germany(物理和天文学系,海德堡大学,海德堡,德国);Interdisciplinary Center for Scientific Computing, Heidelberg University(科学计算跨学科中心,海德堡大学);Hector Institute for Artificial Intelligence in Psychiatry, Central Institute of Mental Health, Mannheim, Germany(精神病学人工智能研究所,心理健康中央研究所,曼海姆,德国);Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany(数学和计算机科学系,海德堡大学,海德堡,德国)
Smooth Piecewise Cutting for Neural Operator to Handle Discontinuities and Sharp Transitions
Mannheim Institute for Intelligent Systems in Medicine, Heidelberg University(海德堡大学曼海姆智能医学研究所);Interdisciplinary Center for Scientific Computing (IWR), Heidelberg University(海德堡大学跨学科科学计算中心);Heidelberg Institute for Theoretical Studies (HITS), Heidelberg University(海德堡大学理论研究 institute);Central Institute for Computer Engineering (ZITI), Heidelberg University(海德堡大学计算机工程中心);CZS Heidelberg Initiative for Model-Based AI (MBAI), Heidelberg University(海德堡模型驱动人工智能倡议)