探索多语言语言模型中各向异性与离群值以用于跨语言语义句子相似度
Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity
AI总结：
本文研究多语言预训练模型的离群维度与各向异性，通过移除离群维、聚类增强和 ZCA 白化等无微调嵌入变换，评估其对跨语言句子相似度的改进。
摘要：
先前工作表明，上下文语言模型输出的表示比静态类型嵌入更具各向异性，并且通常表现出离群维度。这似乎对单语言和多语言模型都成立，尽管在多语言情境下开展的工作要少得多。这些离群值为何出现以及它们如何影响表示，仍是一个活跃的研究领域。我们研究多个预训练多语言语言模型中的离群维度及其与各向异性的关系。我们聚焦跨语言语义相似度任务，因为这些是评估多语言表示的自然任务。具体而言，我们考察句子表示。在平行资源（并非总是可得）上微调的 Sentence transformers 在该任务上表现更好，并且我们表明它们的表示更具各向同性。然而，我们旨在总体上改进多语言表示。我们研究仅通过变换嵌入空间而不进行微调，能够弥补多少性能差距，并可视化所得空间。我们测试了不同操作：移除单个离群维度、基于聚类的各向同性增强以及 ZCA 白化。为便于复现，我们公开了代码。
英文摘要：
Previous work has shown that the representations output by contextual language models are more anisotropic than static type embeddings, and typically display outlier dimensions. This seems to be true for both monolingual and multilingual models, although much less work has been done on the multilingual context. Why these outliers occur and how they affect the representations is still an active area of research. We investigate outlier dimensions and their relationship to anisotropy in multiple pre-trained multilingual language models. We focus on cross-lingual semantic similarity tasks, as these are natural tasks for evaluating multilingual representations. Specifically, we examine sentence representations. Sentence transformers which are fine-tuned on parallel resources (that are not always available) perform better on this task, and we show that their representations are more isotropic. However, we aim to improve multilingual representations in general. We investigate how much of the performance difference can be made up by only transforming the embedding space without fine-tuning, and visualise the resulting spaces. We test different operations: Removing individual outlier dimensions, cluster-based isotropy enhancement, and ZCA whitening. We publish our code for reproducibility.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
