通过正交约束缓解跨语言嵌入中的语义泄漏
Mitigating Semantic Leakage in Cross-lingual Embeddings via Orthogonality Constraint
AI总结：
针对跨语言句子嵌入解耦中存在的语义泄漏问题，提出基于类内聚类与类间分离的ORACLE正交约束训练目标，有效提升跨语言检索与语义相似度任务中的语义对齐效果。
摘要：
在跨语言句子嵌入中准确对齐上下文表示是高效平行数据挖掘的关键。实现这种对齐的常用策略是从多语言预训练模型得到的句子嵌入中解耦语义与语言。然而我们发现，当前的解耦表示学习方法存在语义泄漏问题——我们提出该术语，用于描述大量语言特定信息意外泄漏到语义表示中的现象。这会阻碍语义与语言表示的有效解耦，使得难以检索到能鲜明表征句子含义的嵌入。为解决这一挑战，我们提出一种新型训练目标：正交约束学习（ORthogonAlity Constraint LEarning，ORACLE），专门用于强制语义嵌入与语言嵌入之间的正交性。ORACLE 基于两个组件构建：类内聚类与类间分离。通过在跨语言检索和语义文本相似度任务上开展实验，我们证明采用 ORACLE 目标训练可有效减少语义泄漏，并增强嵌入空间内的语义对齐。
英文摘要：
Accurately aligning contextual representations in cross-lingual sentence embeddings is key for effective parallel data mining. A common strategy for achieving this alignment involves disentangling semantics and language in sentence embeddings derived from multilingual pre-trained models. However, we discover that current disentangled representation learning methods suffer from semantic leakage - a term we introduce to describe when a substantial amount of language-specific information is unintentionally leaked into semantic representations. This hinders the effective disentanglement of semantic and language representations, making it difficult to retrieve embeddings that distinctively represent the meaning of the sentence. To address this challenge, we propose a novel training objective, ORthogonAlity Constraint LEarning (ORACLE), tailored to enforce orthogonality between semantic and language embeddings. ORACLE builds upon two components: intra-class clustering and inter-class separation. Through experiments on cross-lingual retrieval and semantic textual similarity tasks, we demonstrate that training with the ORACLE objective effectively reduces semantic leakage and enhances semantic alignment within the embedding space.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
