MaskLID：通过迭代掩码进行语码转换语言识别
MaskLID: Code-Switching Language Identification through Iterative Masking
AI总结：
MaskLID提出一种无需训练的迭代掩码方法，通过让LID自身识别并掩码主导语言特征，使句子级语言识别器能检测语码转换中的次要语言，并在GlotLID和OpenLID上验证有效性。
摘要：
我们提出MaskLID，一种简单而有效的语码转换（CS）语言识别（LID）方法。MaskLID不需要任何训练，旨在补充当前高性能的句子级LID。句子级LID是在单语文本上训练的分类器，用于提供单一标签，通常使用softmax层将分数转换为概率。然而，当句子同时由L1和L2两种语言组成时，LID分类器往往只返回主导标签L1。为解决这一局限，MaskLID采用一种策略来掩码与L1相关的文本特征，使LID在下一轮将文本分类为L2。该方法利用LID自身来识别需要掩码的特征，不依赖任何外部资源。在这项工作中，我们探索了将MaskLID用于两个开源LID（GlotLID和OpenLID），两者均基于FastText架构。代码和演示可在https://github.com/cisnlp/MaskLID获取。
英文摘要：
We present MaskLID, a simple, yet effective, code-switching (CS) language identification (LID) method. MaskLID does not require any training and is designed to complement current high-performance sentence-level LIDs. Sentence-level LIDs are classifiers trained on monolingual texts to provide single labels, typically using a softmax layer to turn scores into probabilities. However, in cases where a sentence is composed in both L1 and L2 languages, the LID classifier often only returns the dominant label L1. To address this limitation, MaskLID employs a strategy to mask text features associated with L1, allowing the LID to classify the text as L2 in the next round. This method uses the LID itself to identify the features that require masking and does not rely on any external resource. In this work, we explore the use of MaskLID for two open-source LIDs (GlotLID and OpenLID), that are both based on the FastText architecture. Code and demo are available at https://github.com/cisnlp/MaskLID.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
