arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.09190cs.LGcs.AI
收藏

梯度残差连接

Gradient Residual Connections

Department of Engineering Science, University of Oxford(牛津大学工程科学系);Computer Engineering Department, University of Arizona(亚利桑那大学计算机工程系)

Yangchen Pan, Qizhen Ying, Philip Torr, Bo Liu

AI总结:

本文提出梯度残差连接,通过利用梯度信息提升神经网络对高频函数的近似能力,在超分辨率等任务中表现优异。

摘要:

现有研究已将函数梯度的性质与函数近似难度联系起来。受这些见解的启发,我们研究了如何利用梯度信息来提高神经网络近似高频函数的能力,并提出了一种基于梯度的残差连接,作为残差网络中标准身份跳过连接的补充。我们提供了简单的理论直觉,解释了为什么梯度信息可以帮助区分输入并改善具有快速变化行为的函数的近似。在具有高频率正弦波真实值的合成回归任务中,我们显示传统残差连接难以捕捉高频模式。相比之下,我们的梯度残差显著提高了近似质量。然后我们引入了标准残差和梯度残差的凸组合,使网络能够灵活地控制其依赖梯度信息的程度。在通过消融研究验证了我们所提方法的设计选择后,我们进一步在单图像超分辨率任务上验证了我们方法的实用性,其中底层函数可能是高频的。最后,在图像分类和分割等标准任务中,我们的方法性能与标准残差网络相当,表明其广泛的应用价值。

英文摘要:

Existing work has linked properties of a function's gradient to the difficulty of function approximation. Motivated by these insights, we study how gradient information can be leveraged to improve neural network's ability to approximate high-frequency functions, and we propose a gradient-based residual connection as a complement to the standard identity skip connection used in residual networks. We provide simple theoretical intuition for why gradient information can help distinguish inputs and improve the approximation of functions with rapidly varying behaviour. On a synthetic regression task with a high-frequency sinusoidal ground truth, we show that conventional residual connections struggle to capture high-frequency patterns. In contrast, our gradient residual substantially improves approximation quality. We then introduce a convex combination of the standard and gradient residuals, allowing the network to flexibly control how strongly it relies on gradient information. After validating the design choices of our proposed method through an ablation study, we further validate our approach's utility on the single-image super-resolution task, where the underlying function may be high-frequency. Finally, on standard tasks such as image classification and segmentation, our method achieves performance comparable to standard residual networks, suggesting its broad utility.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑