小权值神经网络的代数构造
On Algebraic Constructions of Neural Networks with Small Weights
摘要：
神经门根据输入变量的加权和计算函数。神经门的表达能力（它可以计算的不同函数的数量）取决于权重的大小，一般来说，需要大的权重（输入数量的指数）。在电路复杂性理论和神经计算实践中，研究权值大小、电路大小和深度之间的权衡是一个很好的研究课题。我们提出了一种新的方法来研究这些复杂性权衡考虑相关的代数框架。具体地说，给定一个具有任意系数的线性方程，我们想用一个具有更小（甚至是常数）系数的线性方程组来表达它。我们开发的技术是基于Siegel引理的界限，反浓度不等式的存在性结果和扩展的Sylvester型Hadamard矩阵的建设。我们显式地构造一个恒定权重、最佳大小的矩阵来计算EQUALITY函数（检查以二进制表示的两个整数是否相等）。用一个线性方程计算EQUALITY需要指数级大的权重。此外，我们证明了存在最著名的权重大小（线性）矩阵来计算比较函数（比较两个以二进制表示的整数）。在电路复杂性理论的背景下，我们的研究结果提高了最知名的电路大小的重量大小的上限为平等和比较。
英文摘要：
Neural gates compute functions based on weighted sums of the input variables. The expressive power of neural gates (number of distinct functions it can compute) depends on the weight sizes and, in general, large weights (exponential in the number of inputs) are required. Studying the trade-offs among the weight sizes, circuit size and depth is a well-studied topic both in circuit complexity theory and the practice of neural computation. We propose a new approach for studying these complexity trade-offs by considering a related algebraic framework. Specifically, given a single linear equation with arbitrary coefficients, we would like to express it using a system of linear equations with smaller (even constant) coefficients. The techniques we developed are based on Siegel's Lemma for the bounds, anti-concentration inequalities for the existential results and extensions of Sylvester-type Hadamard matrices for the constructions. We explicitly construct a constant weight, optimal size matrix to compute the EQUALITY function (checking if two integers expressed in binary are equal). Computing EQUALITY with a single linear equation requires exponentially large weights. In addition, we prove the existence of the best-known weight size (linear) matrices to compute the COMPARISON function (comparing between two integers expressed in binary). In the context of the circuit complexity theory, our results improve the upper bounds on the weight sizes for the best-known circuit sizes for EQUALITY and COMPARISON.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
