arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.18540cs.CC
收藏

二进制输入的ReLU网络的深度下界

Depth Lower Bounds for ReLU Networks with Binary Inputs

Neil Krishnan, Elchanan Mossel

AI总结:

针对二进制输入和实值输出的ReLU网络,构造了一个深度n+1、宽度常数的函数族,证明任何深度d、宽度w的精确计算网络需满足w^d = Ω(2^n),即深度d = o(n/log n)时宽度不能为n的多项式。

摘要:

我们研究了具有离散(布尔)输入和实值输出的ReLU网络中深度的作用,补充了两个已有的研究方向。对于布尔输入,在$\mathsf{AC}^0$中证明了显著的深度分离结果,但使用阈值($\mathsf{TC}^0$)或ReLU门时,深度分离仅针对深度二与三建立。另一方面,对于{\em实值}函数和ReLU网络,Telgarsky(2016)构造了一个简单的单变量函数类,在更高深度上建立了分离。本文旨在为$\{0,1\}^n$上的ReLU网络建立全深度分离。我们通过展示一个显式的函数族来实现这一点,该函数族可由深度$n+1$、宽度常数的ReLU网络精确计算,而任何深度$d$、宽度$w$的ReLU网络要精确计算该函数必须满足$w^d = \Omega(2^n)$;特别地,没有深度$d = o(n/\log n)$的网络可以用$n$的多项式宽度计算它。我们注意到,我们的下界依赖于\emph{精确、无限精度}计算,因为输出的指数精度截断可由多项式大小的$\mathsf{TC}^0$电路计算。

英文摘要:

We study the role of depth in ReLU networks with discrete (Boolean) inputs and real-valued outputs, complementing two established lines of work. For Boolean inputs, striking depth separation results were proven for $\mathsf{AC}^0$ but with threshold ($\mathsf{TC}^0$) or ReLU gates depth separation is only established for depth two vs. three. On the other hand, for {\em real-valued} functions and ReLU networks, Telgarsky's (2016) constructed a simple one variable class of functions which establishes separation at higher depths. In this paper we are interested to establish an all-depths depth separation for ReLU networks on $\{0,1\}^n$. We do so by exhibiting an explicit family of functions computable exactly by a ReLU network of depth $n+1$ and constant width, such that any ReLU network of depth $d$ and width $w$ computing the function exactly must satisfy $w^d = Ω(2^n)$; in particular, no network of depth $d = o(n/\log n)$ can compute it with width polynomial in $n$. We note that our lower bound relies on \emph{exact, infinite-accuracy} computation as an exponential precision truncation of the output is computable by a polynomial-size $\mathsf{TC}^0$ circuit.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑