自监督Transformer中的分离归一化研究
On Separate Normalization in Self-supervised Transformers
AI总结：
本文提出为自监督Transformer中的token和[CLS]符号分别设置独立归一化层，以更好捕捉各自特征，并在图像、自然语言和图任务上平均提升2.7%性能。
摘要：
面向Transformer的自监督训练方法已在多个领域展现出卓越性能。以往基于Transformer的模型，例如掩码自编码器（MAE），通常对[CLS]符号和其余token使用同一个归一化层。本文提出一种简单修改：对token和[CLS]符号分别使用独立的归一化层，以更好地捕捉二者不同的特征并提升下游任务性能。我们的方法旨在缓解对两类token使用相同归一化统计量可能带来的负面影响，因为同一统计量未必与它们各自的作用最优对齐。实验表明，采用分离归一化层后，[CLS]嵌入能够更好地编码全局上下文信息，并在其各向异性空间中分布得更加均匀。将传统归一化层替换为两个分离归一化层后，我们在图像、自然语言和图三个领域上观察到平均2.7%的性能提升。
英文摘要：
Self-supervised training methods for transformers have demonstrated remarkable performance across various domains. Previous transformer-based models, such as masked autoencoders (MAE), typically utilize a single normalization layer for both the [CLS] symbol and the tokens. We propose in this paper a simple modification that employs separate normalization layers for the tokens and the [CLS] symbol to better capture their distinct characteristics and enhance downstream task performance. Our method aims to alleviate the potential negative effects of using the same normalization statistics for both token types, which may not be optimally aligned with their individual roles. We empirically show that by utilizing a separate normalization layer, the [CLS] embeddings can better encode the global contextual information and are distributed more uniformly in its anisotropic space. When replacing the conventional normalization layer with the two separate layers, we observe an average 2.7% performance improvement over the image, natural language, and graph domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
