arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2112.11577cs.LG
收藏

坐标MLP的位置嵌入学习

Learning Positional Embeddings for Coordinate-MLPs

Australian Institute for Machine Learning(澳大利亚机器学习研究所);University of Adelaide(阿德莱德大学)

Sameera Ramasinghe, Simon Lucey

摘要:

我们提出了一种新的方法来提高性能的坐标MLP通过学习特定实例的位置嵌入。位置嵌入参数和网络权重的端到端优化导致泛化性能差。相反,我们开发了一个通用的框架来学习基于经典图拉普拉斯正则化的位置嵌入,它可以隐含地平衡记忆和泛化之间的权衡。然后,这个框架被用来提出一种新的位置嵌入方案,其中每个坐标(即实例)学习超参数以提供最佳性能。我们表明,建议的嵌入实现更好的性能,更高的稳定性相比,建立良好的随机傅立叶特征(RFF)。此外,我们证明了所提出的嵌入方案产生稳定的梯度,使无缝集成到深层架构作为中间层。

英文摘要:

We propose a novel method to enhance the performance of coordinate-MLPs by learning instance-specific positional embeddings. End-to-end optimization of positional embedding parameters along with network weights leads to poor generalization performance. Instead, we develop a generic framework to learn the positional embedding based on the classic graph-Laplacian regularization, which can implicitly balance the trade-off between memorization and generalization. This framework is then used to propose a novel positional embedding scheme, where the hyperparameters are learned per coordinate (i.e, instance) to deliver optimal performance. We show that the proposed embedding achieves better performance with higher stability compared to the well-established random Fourier features (RFF). Further, we demonstrate that the proposed embedding scheme yields stable gradients, enabling seamless integration into deep architectures as intermediate layers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑