利用快速傅里叶变换高效学习去相关表示
Learning Decorrelated Representations Efficiently Using Fast Fourier Transform
AI总结：
本文针对 Barlow Twins 和 VICReg 在高维嵌入下训练开销大的问题，提出一种基于快速傅里叶变换的松弛去相关正则化器，将计算复杂度从 O(n d^2) 降至 O(n d log d)，并缓解局部极小值问题，在保持相当准确率的同时降低内存占用并提升大维度下的训练速度。
摘要：
Barlow Twins 和 VICReg 是使用正则化器来对特征进行去相关的自监督表示学习模型。尽管这些模型与传统的表示学习模型同样有效，但如果投影嵌入的维度 d 较高，其训练可能在计算上要求很高。由于正则化器是根据互相关矩阵或协方差矩阵的各个元素定义的，因此计算 n 个样本的损失需要 O(n d^2) 时间。在本文中，我们提出了一种松弛的去相关正则化器，它可以通过快速傅里叶变换在 O(n d log d) 时间内计算。我们还提出了一种低成本技术，以缓解随着松弛而出现的不良局部极小值。所提出的正则化器在下游任务中表现出与现有正则化器相当的准确率，而其训练所需的内存更少，并且在 d 较大时速度更快。源代码已公开。
英文摘要：
Barlow Twins and VICReg are self-supervised representation learning models that use regularizers to decorrelate features. Although these models are as effective as conventional representation learning models, their training can be computationally demanding if the dimension d of the projected embeddings is high. As the regularizers are defined in terms of individual elements of a cross-correlation or covariance matrix, computing the loss for n samples takes O(n d^2) time. In this paper, we propose a relaxed decorrelating regularizer that can be computed in O(n d log d) time by Fast Fourier Transform. We also propose an inexpensive technique to mitigate undesirable local minima that develop with the relaxation. The proposed regularizer exhibits accuracy comparable to that of existing regularizers in downstream tasks, whereas their training requires less memory and is faster for large d. The source code is available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
