通过逐层维度选择从预训练语言模型中拆解单词语义
Breaking Down Word Semantics from Pre-trained Language Models through Layer-wise Dimension Selection
AI总结：
针对预训练语言模型嵌入中语义等维度难以解读的问题，本文提出在不更新预训练参数的前提下，通过对BERT各层中间输出施加二值掩码解耦语义义项的方法，实验验证了逐层信息的有效性及语义解耦对词义判别性能的提升作用。
摘要：
从预训练语言模型（PLM）中获取的上下文词嵌入已被证明在各类词级自然语言处理任务中效果显著。然而，解读嵌入中潜藏的句法、语义等维度仍颇具挑战。解耦表示学习已成为一种颇具前景的方法，可将特定维度分离为独立的嵌入。此外，学界认为不同的语言知识存储在PLM的不同层中。本文旨在通过对各层中间输出施加二值掩码，在不更新预训练参数的前提下，从BERT中解耦出语义义项。研究通过二分类任务评估解耦后的嵌入，判断两个不同句子中的目标词是否具有相同含义。在大小写敏感的BERT_base上的实验表明，利用逐层信息是有效的，且解耦语义义项可进一步提升性能。
英文摘要：
Contextual word embeddings obtained from pre-trained language model (PLM) have proven effective for various natural language processing tasks at the word level. However, interpreting the hidden aspects within embeddings, such as syntax and semantics, remains challenging. Disentangled representation learning has emerged as a promising approach, which separates specific aspects into distinct embeddings. Furthermore, different linguistic knowledge is believed to be stored in different layers of PLM. This paper aims to disentangle semantic sense from BERT by applying a binary mask to middle outputs across the layers, without updating pre-trained parameters. The disentangled embeddings are evaluated through binary classification to determine if the target word in two different sentences has the same meaning. Experiments with cased BERT$_{\texttt{base}}$ show that leveraging layer-wise information is effective and disentangling semantic sense further improve performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
