arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.13291cs.CVcs.GR
收藏

将局部化高斯函数作为自注意力权重用于点云匹配

Localized Gaussians as Self-Attention Weights for Point Clouds Correspondence

University of Milano-Bicocca, Department of Informatics, Systems and Communication(米兰-比科卡大学信息与通信系统学院)

Alessandro Riva, Alessandro Raganato, Simone Melzi

AI总结:

研究将局部化高斯函数作为固定注意力权重整合到Transformer架构中用于点云匹配,发现该方法能加速训练、增强优化稳定性,并通过消融实验确定了注入信息最有效的层级。

摘要:

当前数据驱动的点云匹配方法需要大量训练时间和计算资源,给模型部署和应用带来了重大挑战。在点云匹配任务中,近期基于仅编码器 Transformer 架构的研究进展揭示了注意力头中出现了具有语义意义的模式,特别类似于以输入形状每个点为中心的高斯函数。在本研究中,我们通过将这些模式作为固定的注意力权重整合到 Transformer 架构的注意力头中来进一步探究这一现象。我们评估了两种变体:一种利用高斯函数的预定方差值,另一种将方差值视为可学习参数。此外,我们分析了在噪声数据上的性能,并探索了提高噪声鲁棒性的可能方法。我们的研究结果表明,固定注意力权重不仅加速了训练过程,还增强了优化的稳定性。另外,我们进行了消融研究,以确定注入信息影响最大的特定层,并理解网络对该信息的依赖性。

英文摘要:

Current data-driven methodologies for point cloud matching demand extensive training time and computational resources, presenting significant challenges for model deployment and application. In the point cloud matching task, recent advancements with an encoder-only Transformer architecture have revealed the emergence of semantically meaningful patterns in the attention heads, particularly resembling Gaussian functions centered on each point of the input shape. In this work, we further investigate this phenomenon by integrating these patterns as fixed attention weights within the attention heads of the Transformer architecture. We evaluate two variants: one utilizing predetermined variance values for the Gaussians, and another where the variance values are treated as learnable parameters. Additionally we analyze the performances on noisy data and explore a possible way to improve robustness to noise. Our findings demonstrate that fixing the attention weights not only accelerates the training process but also enhances the stability of the optimization. Furthermore, we conducted an ablation study to identify the specific layers where the infused information is most impactful and to understand the reliance of the network on this information.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑