语码转换语言识别比你想象的更难
Code-Switched Language Identification is Harder Than You Think
AI总结:
本研究聚焦语码转换语言识别任务,将其重构为句子级多标签标注问题,测试三种模型并提出更适配的评估指标,发现现有方法均无法充分胜任该任务,为后续研究提供了方向。
摘要:
语码转换(CS)是书面和口语交流中非常普遍的现象,但许多自然语言处理应用对其处理效果不佳。面向构建语码转换语料库的应用场景,我们探索用于语料库构建的CS语言识别(LID)任务。我们将任务扩展到更多语言,并考虑采用架构更简单、推理速度更快的模型,让任务设定更贴合实际。我们还将该任务重新定义为句子级多标签标注问题,以降低求解难度。在明确任务设定后,我们研究了三种适用于该任务的合理模型,并定义了更能反映理想性能的评估指标。我们通过实验证据表明,目前没有任何方法能充分胜任该任务,最后为该领域的未来研究提出了建议。
英文摘要:
Code switching (CS) is a very common phenomenon in written and spoken communication but one that is handled poorly by many natural language processing applications. Looking to the application of building CS corpora, we explore CS language identification (LID) for corpus building. We make the task more realistic by scaling it to more languages and considering models with simpler architectures for faster inference. We also reformulate the task as a sentence-level multi-label tagging problem to make it more tractable. Having defined the task, we investigate three reasonable models for this task and define metrics which better reflect desired performance. We present empirical evidence that no current approach is adequate and finally provide recommendations for future work in this area.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。