通过精确线性代数实现更浅的ReLU网络表示

中文摘要 AI 辅助

我们证明，对于每个\(n\leq10\)，\(n\)个实数的最大值可由具有两个隐藏层的ReLU网络精确表示。通过将问题简化为精确的有理线性代数来进行构造：经过对称约简后，必要的抵消在\(\mathbb{Q}\)上的有限线性系统中编码，我们通过计算求解并验证。\(\max_{10}\)的表示在第一个隐藏层具有仅由成对最大值组成的结构，这一特征使其能够递归地代入更大的网络。我们据此表明，对于每个\(n>10\)，最大值\(\max_{n}\)可以用\(\lceil{\log_5 (n / 2)\rceil}+1 < \log_5(n) +1.5694\)个隐藏层精确表示。通过广义铰链超平面表示，对于\(\mathbb{R}^d\)上的所有连续分段线性函数，同样的深度界限成立，这里用\(d + 1\)代替\(n\)。特别地，对于\(d\leq9\)的\(\mathbb{R}^d\)上的每个连续分段线性函数都允许有一个双隐藏层ReLU表示。我们的结果改进了[Bakaev, Brunck, Hertrich, Stade, Yehudayoff, STOC'26]中的工作。在该工作中，作者为\(\max_{5}\)建立了双隐藏层表示，为\(\max_{n}\)建立了\(\lceil{\log_3 (n - 2)\rceil}+1\)个隐藏层的上限。

英文摘要

We study the depth required by ReLU networks to exactly represent piecewise linear functions, focusing specifically on the maximum function. This problem has recently received significant attention in both the ML and TCS literature. We prove that $\max_n(x)=\max\{x_1,\ldots,x_n\}$ is exactly representable with two hidden layers for every $n\leq 12$. Previously, this was only known up to $n\leq5$ [Bakaev, Brunck, Hertrich, Stade, Yehudayoff, STOC'26]. We obtain our constructions through an exact computer-assisted search within a space of candidate solutions: After a symmetry reduction, we obtain a finite system of linear equations over $\mathbb{Q}$ such that any solution yields a valid representation of the maximum function. The resulting constructions have a structured first hidden layer, which enables recursive substitution into deeper networks. This yields an exact ReLU representation of $\max_n$ with at most $\lceil \log_6(n/2) \rceil+1$ hidden layers. Consequently, every continuous piecewise-linear function on $\mathbb{R}^d$ admits an exact representation with at most $\lceil\log_6((d+1)/2)\rceil+1$ hidden layers; in particular, two hidden layers suffice for $d\leq 11$. Again, these results improve upon [Bakaev, Brunck, Hertrich, Stade, Yehudayoff, STOC'26], who proved analogous logarithmic bounds with base three.

发表机构

• University of Technology Nuremberg（纽伦堡工业大学）
• Brandenburg University of Technology Cottbus–Senftenberg（科特布斯-森夫滕贝格勃兰登堡工业大学）
• University of Copenhagen（哥本哈根大学）
• Max Planck Institute for Mathematics in the Sciences（马克斯·普朗克科学数学研究所）
• Freie Universität Berlin（柏林自由大学）

机构由 AI 辅助整理，请以论文原文为准。