BERT-JEPA: 重新组织CLS嵌入以实现语言无关语义
BERT-JEPA: Reorganizing CLS Embeddings for Language-Invariant Semantics
AI总结:
BERT-JEPA通过引入JEPA训练目标,改进BERT模型以解决CLS嵌入空间坍塌问题,提升多语言基准性能。
摘要:
联合嵌入预测架构(JEPA)是一种新颖的自监督训练技术,已在多个领域展示了近期的潜力。我们引入BERT-JEPA(BEPA),一种将JEPA训练目标添加到BERT风格模型中的训练范式,旨在对抗坍塌的[CLS]嵌入空间,并将其转换为语言无关的空间。这种新结构在多语言基准测试中带来了性能的提升。
英文摘要:
Joint Embedding Predictive Architectures (JEPA) are a novel self supervised training technique that have shown recent promise across domains. We introduce BERT-JEPA (BEPA), a training paradigm that adds a JEPA training objective to BERT-style models, working to combat a collapsed [CLS] embedding space and turning it into a language-agnostic space. This new structure leads to increased performance across multilingual benchmarks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。