arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.16601cs.CLcs.AI
收藏

MixLID:混合语言文本中的令牌级语言识别

MixLID: Token-Level Language Identification in Mixed-Language Text

Korea University(高丽大学)

Seongtae Hong, Youngjoon Jang, Seungyoon Lee, Jungseob Lee, Heuiseok Lim

AI总结:

本文提出支持111种语言-文字组合的MixLID令牌级语言识别模型,采用两阶段训练流程,在单语和混合语言识别任务中均优于基准模型,性能优异且可公开获取。

摘要:

处理多语言文本需要识别广泛的语言,并在包含多种语言的输入中区分每个令牌的语言。我们提出MixLID,这是一个支持111种语言-文字组合的令牌级语言识别模型。MixLID采用两阶段训练流程:首先使用单语示例的语言标签学习区分单个语言,然后在混合语言数据上继续训练。第二阶段使用现有代码切换数据的串联示例,以及基于英语源词与借词对应关系通过单词替换创建的增强示例。在单语识别任务中,在与每个基准共享的语言覆盖范围内的四个基准上,MixLID在准确率和宏F1值上均优于所有基准模型;在混合语言识别任务中,在所有评估的语言对上,它在识别单个令牌语言的性能上优于基准模型。MixLID在单语和混合语言场景下的广泛语言中均实现了高性能,该模型可在指定URL获取。

英文摘要:

Processing multilingual text requires both identifying a broad range of languages and distinguishing the language of each token in inputs containing multiple languages. We introduce MixLID, a token-level language identification model supporting 111 language-script combinations. MixLID follows a two-stage training process: it first learns to distinguish individual languages using the language labels of monolingual examples, then continues training on mixed-language data. The second stage uses concatenated examples from existing code-switching data alongside augmented examples created through word substitution based on correspondences between English source words and loanwords. For monolingual identification, MixLID outperforms all baselines in both accuracy and macro F1 on four benchmarks within the language coverage shared with each baseline. For mixed-language identification, it outperforms the baselines in identifying the language of individual tokens on every evaluated language pair. MixLID achieves high performance across a broad range of languages in both monolingual and mixed-language settings. The model is publicly available at https://huggingface.co/nlpai-lab/MixLID.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑