在坐标网络中权衡位置复杂度与深度
Trading Positional Complexity vs. Deepness in Coordinate Networks
AI总结:
本文研究了坐标网络中位置编码的复杂度与深度的权衡,提出更一般的理论分析方法,并通过实验验证了更复杂的编码可通过线性网络实现优异性能。
摘要:
已知基于坐标的MLP通过将坐标位置编码为傅里叶特征数组,在保留高频信息方面获益。此前,这些位置编码的有效性主要通过傅里叶视角研究。本文旨在拓宽这一理解,证明替代的非傅里叶嵌入函数也可用于位置编码。我们进一步表明,其性能完全由嵌入矩阵的稳定秩与嵌入坐标间距离的权衡决定。我们还证明现在普遍使用的傅里叶特征映射是特殊情况,满足这些条件。因此,我们提出更一般的理论来分析基于移位基函数的位置编码。此外,我们论证使用更复杂的编码(其复杂度随模式数指数增长)仅需线性(而非深层)坐标函数即可实现相当性能。反直觉的是,我们证明用位置编码复杂度换取网络深度的速度比当前最先进的方法快多个数量级;尽管有额外的编码复杂度。为此,我们开发了必要的理论公式,并通过实验证明我们的理论主张在实践中成立。
英文摘要:
It is well noted that coordinate-based MLPs benefit -- in terms of preserving high-frequency information -- through the encoding of coordinate positions as an array of Fourier features. Hitherto, the rationale for the effectiveness of these positional encodings has been mainly studied through a Fourier lens. In this paper, we strive to broaden this understanding by showing that alternative non-Fourier embedding functions can indeed be used for positional encoding. Moreover, we show that their performance is entirely determined by a trade-off between the stable rank of the embedded matrix and the distance preservation between embedded coordinates. We further establish that the now ubiquitous Fourier feature mapping of position is a special case that fulfills these conditions. Consequently, we present a more general theory to analyze positional encoding in terms of shifted basis functions. In addition, we argue that employing a more complex positional encoding -- that scales exponentially with the number of modes -- requires only a linear (rather than deep) coordinate function to achieve comparable performance. Counter-intuitively, we demonstrate that trading positional embedding complexity for network deepness is orders of magnitude faster than current state-of-the-art; despite the additional embedding complexity. To this end, we develop the necessary theoretical formulae and empirically verify that our theoretical claims hold in practice.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。