各向异性是Transformer固有的特性吗？
Is Anisotropy Inherent to Transformers?
AI总结：
该研究针对Transformer表征退化的各向异性问题，通过实验验证特定目标语言模型及多模态Transformer中均存在该现象，提出各向异性可能是Transformer类模型的固有属性，突破了此前仅归因于长尾分布交叉熵损失优化的认知。
摘要：
表征退化问题是在基于Transformer的自监督学习方法中被广泛观察到的现象。在自然语言处理（NLP）领域，它表现为各向异性，即隐藏表征的一种特殊属性，使得它们在角距离（余弦相似度）层面出人意料地彼此接近。近期部分研究表明，各向异性是在长尾分布的词元上优化交叉熵损失带来的结果。本文通过实验证明，在具备特定目标、本不应直接受该问题影响的语言模型中，同样能观察到各向异性现象。我们还发现，各向异性问题存在于在其他模态上训练的Transformer中。这些观察结果表明，各向异性实际上可能是基于Transformer的模型所固有的特性。
英文摘要：
The representation degeneration problem is a phenomenon that is widely observed among self-supervised learning methods based on Transformers. In NLP, it takes the form of anisotropy, a singular property of hidden representations which makes them unexpectedly close to each other in terms of angular distance (cosine-similarity). Some recent works tend to show that anisotropy is a consequence of optimizing the cross-entropy loss on long-tailed distributions of tokens. We show in this paper that anisotropy can also be observed empirically in language models with specific objectives that should not suffer directly from the same consequences. We also show that the anisotropy problem extends to Transformers trained on other modalities. Our observations tend to demonstrate that anisotropy might actually be inherent to Transformers-based models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
