LGSE: 词法基础的子词嵌入初始化用于低资源语言适应
LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation
AI总结：
LGSE通过引入词法信息的子词分割方法，改进低资源语言适应中的词嵌入初始化，通过平均预训练的子词或FastText词法表示，提升表示质量。
摘要：
适应预训练语言模型到低资源、形态丰富的语言仍是一个重大挑战。现有词汇扩展方法通常依赖任意分割的子词单元，导致词汇表示碎片化并丢失关键形态学信息。为解决这一限制，我们提出了词法基础的子词嵌入初始化（LGSE）框架，该框架引入了具有形态学信息的分割方法来初始化新词的嵌入。而不是使用随机向量或任意子词，LGSE将词分解为构成的词素，并通过平均预训练的子词或FastText基于的词素表示构建语义连贯的嵌入。当一个词无法分割成有意义的词素时，其嵌入通过字符n-gram表示构建以捕捉结构信息。在语言适应预训练中，我们应用一个正则化项，惩罚新引入的嵌入与初始化值的大幅偏离，从而保持与原始预训练嵌入空间的一致性，同时使模型适应目标语言。为了隔离初始化的影响，我们保留原始预训练模型的词汇和分词器，并在适应过程中仅更新新的嵌入。我们评估LGSE在三个NLP任务：问答、命名实体识别和文本分类上，于两种形态丰富、低资源语言：阿姆哈拉语和提格里尼亚语，其中形态分割资源可用。实验结果表明，LGSE在所有任务上均优于基线方法，证明了词法基础嵌入初始化在提升低资源语言表示质量方面的有效性。项目资源可在GitHub链接中获得。
英文摘要：
Adapting pretrained language models to low-resource, morphologically rich languages remains a significant challenge. Existing vocabulary expansion methods typically rely on arbitrarily segmented subword units, resulting in fragmented lexical representations and loss of critical morphological information. To address this limitation, we propose the Lexically Grounded Subword Embedding Initialization (LGSE) framework, which introduces morphologically informed segmentation for initializing embeddings of novel tokens. Instead of using random vectors or arbitrary subwords, LGSE decomposes words into their constituent morphemes and constructs semantically coherent embeddings by averaging pretrained subword or FastText-based morpheme representations. When a token cannot be segmented into meaningful morphemes, its embedding is constructed using character n-gram representations to capture structural information. During Language-Adaptive Pretraining, we apply a regularization term that penalizes large deviations of newly introduced embeddings from their initialized values, preserving alignment with the original pretrained embedding space while enabling adaptation to the target language. To isolate the effect of initialization, we retain the original pre-trained model vocabulary and tokenizer and update only the new embeddings during adaptation. We evaluate LGSE on three NLP tasks: Question Answering, Named Entity Recognition, and Text Classification, in two morphologically rich, low-resource languages: Amharic and Tigrinya, where morphological segmentation resources are available. Experimental results show that LGSE consistently outperforms baseline methods across all tasks, demonstrating the effectiveness of morphologically grounded embedding initialization for improving representation quality in underrepresented languages. Project resources are available in the GitHub link.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
