面向自监督学习的矩阵信息论
Matrix Information Theory for Self-Supervised Learning
AI总结：
本研究提出基于矩阵信息论的自监督学习方法Matrix-SSL，通过矩阵均匀性损失与对齐损失增强最大熵编码，在图像任务和语言建模任务上均取得优于现有方法的表现。
摘要：
最大熵编码框架为SimSiam、Barlow Twins和MEC等众多非对比学习方法提供了统一视角。受该框架启发，我们提出了Matrix-SSL这一全新方法，它利用矩阵信息论将最大熵编码损失解释为矩阵均匀性损失。此外，Matrix-SSL通过无缝融入矩阵对齐损失来增强最大熵编码方法，可直接对齐不同分支中的协方差矩阵。实验结果表明，在ImageNet数据集的线性评估设置以及MS-COCO的迁移学习任务上，Matrix-SSL的表现均优于现有最优方法。具体而言，在MS-COCO上执行迁移学习任务时，我们的方法仅用400轮预训练（对比现有方法的800轮预训练），性能就超过MoCo v2、BYOL等此前的最优方法最多3.3%。我们还尝试将表示学习引入语言建模领域，使用矩阵交叉熵损失微调一个7B规模的模型，在GSM8K数据集上相比标准交叉熵损失有3.1%的提升。代码可在https://github.com/yifanzhang-pro/Matrix-SSL获取。
英文摘要：
The maximum entropy encoding framework provides a unified perspective for many non-contrastive learning methods like SimSiam, Barlow Twins, and MEC. Inspired by this framework, we introduce Matrix-SSL, a novel approach that leverages matrix information theory to interpret the maximum entropy encoding loss as matrix uniformity loss. Furthermore, Matrix-SSL enhances the maximum entropy encoding method by seamlessly incorporating matrix alignment loss, directly aligning covariance matrices in different branches. Experimental results reveal that Matrix-SSL outperforms state-of-the-art methods on the ImageNet dataset under linear evaluation settings and on MS-COCO for transfer learning tasks. Specifically, when performing transfer learning tasks on MS-COCO, our method outperforms previous SOTA methods such as MoCo v2 and BYOL up to 3.3% with only 400 epochs compared to 800 epochs pre-training. We also try to introduce representation learning into the language modeling regime by fine-tuning a 7B model using matrix cross-entropy loss, with a margin of 3.1% on the GSM8K dataset over the standard cross-entropy loss. Code available at https://github.com/yifanzhang-pro/Matrix-SSL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
