MirrorWiC：从预训练的语言模型中提取词的上下文表示
MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models
摘要：
最近的研究表明，预训练的语言模型（PLM），如BERT和RoBERTa，甚至可以通过简单的自监督技术转换为有效的句子和单词编码器。受这一系列工作的启发，在本文中，我们提出了一种完全无监督的方法来改善PLM中的词上下文（WiC）表示，通过一个简单而有效的WiC目标微调过程实现：WiC WiC。所提出的方法仅利用从维基百科采样的原始文本，假设没有意义注释的数据，并在标准的对比学习设置中学习上下文感知的单词表示。我们实验了一系列跨多种语言的标准和全面的WiC基准测试。我们提出的完全无监督的MPLWiC模型在所有单语，多语言和跨语言设置中获得了比现成PLM更大的收益。此外，在一些标准的WiC基准测试中，WiC甚至可以与使用任务内数据和感觉标签进行微调的监督模型相提并论。
英文摘要：
Recent work indicated that pretrained language models (PLMs) such as BERT and RoBERTa can be transformed into effective sentence and word encoders even via simple self-supervised techniques. Inspired by this line of work, in this paper we propose a fully unsupervised approach to improving word-in-context (WiC) representations in PLMs, achieved via a simple and efficient WiC-targeted fine-tuning procedure: MirrorWiC. The proposed method leverages only raw texts sampled from Wikipedia, assuming no sense-annotated data, and learns context-aware word representations within a standard contrastive learning setup. We experiment with a series of standard and comprehensive WiC benchmarks across multiple languages. Our proposed fully unsupervised MirrorWiC models obtain substantial gains over off-the-shelf PLMs across all monolingual, multilingual and cross-lingual setups. Moreover, on some standard WiC benchmarks, MirrorWiC is even on-par with supervised models fine-tuned with in-task data and sense labels.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
