Sobolev空间中神经网络集的非闭性
Nonclosedness of Sets of Neural Networks in Sobolev Spaces
摘要：
我们研究了Sobolev空间中固定结构的已实现神经网络的闭性。对于一个精确的$m$ -倍可微激活函数$ρ$，我们构造了一个神经网络序列$(Φ_n)_{n \in \mathbb{N}}$，其实现按顺序- $(m-1)$ Sobolev范数收敛到一个不能被神经网络精确实现的函数.因此，实现的神经网络的集合不是按顺序闭合的- $(m-1)$ Sobolev空间$W^{m-1,p}$对于$p \in [1,\infty]$。我们进一步表明，这些集是不封闭的$W^{m,p}$在略强的条件下，对$m$ -th衍生物的$ρ$。对于一个真正的解析激活函数，我们证明了实现的神经网络集是不封闭的$W^{k,p}$的任何$k \in \mathbb{N}$。非封闭性允许近似的非网络目标函数与无界参数增长。我们部分特征的参数增长率为大多数激活函数的实现神经网络的特定序列可以近似激活函数的导数与权重增加成反比的$L^p$近似误差。最后，我们提出的实验结果表明，网络能够密切接近非网络的目标函数，通过训练增加参数。
英文摘要：
We examine the closedness of sets of realized neural networks of a fixed architecture in Sobolev spaces. For an exactly $m$-times differentiable activation function $ρ$, we construct a sequence of neural networks $(Φ_n)_{n \in \mathbb{N}}$ whose realizations converge in order-$(m-1)$ Sobolev norm to a function that cannot be realized exactly by a neural network. Thus, sets of realized neural networks are not closed in order-$(m-1)$ Sobolev spaces $W^{m-1,p}$ for $p \in [1,\infty]$. We further show that these sets are not closed in $W^{m,p}$ under slightly stronger conditions on the $m$-th derivative of $ρ$. For a real analytic activation function, we show that sets of realized neural networks are not closed in $W^{k,p}$ for any $k \in \mathbb{N}$. The nonclosedness allows for approximation of non-network target functions with unbounded parameter growth. We partially characterize the rate of parameter growth for most activation functions by showing that a specific sequence of realized neural networks can approximate the activation function's derivative with weights increasing inversely proportional to the $L^p$ approximation error. Finally, we present experimental results showing that networks are capable of closely approximating non-network target functions with increasing parameters via training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
