arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.01274cs.LG
收藏

随机权重因子分解改进了连续神经表示的训练

Random Weight Factorization Improves the Training of Continuous Neural Representations

University of Pennsylvania(宾夕法尼亚大学)

Sifan Wang, Hanwen Wang, Jacob H. Seidman, Paris Perdikaris

摘要:

连续神经表示最近出现了一个强大的和灵活的替代经典的离散表示的信号。然而,训练它们捕捉多尺度信号中的精细细节是困难的,并且计算成本很高。在这里,我们提出了随机权重分解作为一个简单的插入式替代,用于参数化和初始化基于坐标的多层感知器(MLP)中的传统线性层,从而显着加速和改善其训练。我们展示了这种分解如何改变潜在的损失景观,并有效地使网络中的每个神经元使用自己的自适应学习率进行学习。这不仅有助于减轻频谱偏差,而且还允许网络从糟糕的初始化中快速恢复,并达到更好的局部最小值。我们展示了如何利用随机权重因子分解来改善各种任务的神经表示训练,包括图像回归,形状表示,计算机断层扫描,逆渲染,求解偏微分方程和函数空间之间的学习运算符。

英文摘要:

Continuous neural representations have recently emerged as a powerful and flexible alternative to classical discretized representations of signals. However, training them to capture fine details in multi-scale signals is difficult and computationally expensive. Here we propose random weight factorization as a simple drop-in replacement for parameterizing and initializing conventional linear layers in coordinate-based multi-layer perceptrons (MLPs) that significantly accelerates and improves their training. We show how this factorization alters the underlying loss landscape and effectively enables each neuron in the network to learn using its own self-adaptive learning rate. This not only helps with mitigating spectral bias, but also allows networks to quickly recover from poor initializations and reach better local minima. We demonstrate how random weight factorization can be leveraged to improve the training of neural representations on a variety of tasks, including image regression, shape representation, computed tomography, inverse rendering, solving partial differential equations, and learning operators between function spaces.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑