arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2103.05552cs.CLcs.AIcs.LG
收藏

达罗毗荼语识别方法比较

Comparing Approaches to Dravidian Language Identification

University of Helsinki(赫尔辛基大学);Rochester Institute of Technology(罗切斯特理工学院);University of Wolverhampton(伍尔弗汉普顿大学)

Tommi Jauhiainen, Tharindu Ranasinghe, Marcos Zampieri

摘要:

本文介绍了HWR团队在VarDial 2021研讨会上组织的德拉威语言识别(DLI)共享任务的提交内容。DLI训练集包括16,674条以罗马脚本编写的YouTube评论,其中包含与英语和三种南达罗毗荼语之一(卡纳达语,马拉雅拉姆语和泰米尔语)混合的代码文本。我们提交了使用两种模型生成的结果,一种是具有自适应语言模型的朴素贝叶斯分类器,它在许多语言和方言识别任务中表现出具有竞争力的性能,另一种是基于transformer的模型,它被广泛认为是许多NLP任务中最先进的模型。我们的第一次提交是在封闭的提交轨道中发送的,只使用共享任务组织者提供的训练集,而第二次提交被认为是开放的,因为它使用了一个用外部数据训练的预训练模型。我们的团队在基于朴素贝叶斯的提交中获得了共享任务的第二名。我们的研究结果强化了这样一种观点,即深度学习方法在语言识别相关任务中的竞争力不如在许多其他文本分类任务中的竞争力。

英文摘要:

This paper describes the submissions by team HWR to the Dravidian Language Identification (DLI) shared task organized at VarDial 2021 workshop. The DLI training set includes 16,674 YouTube comments written in Roman script containing code-mixed text with English and one of the three South Dravidian languages: Kannada, Malayalam, and Tamil. We submitted results generated using two models, a Naive Bayes classifier with adaptive language models, which has shown to obtain competitive performance in many language and dialect identification tasks, and a transformer-based model which is widely regarded as the state-of-the-art in a number of NLP tasks. Our first submission was sent in the closed submission track using only the training set provided by the shared task organisers, whereas the second submission is considered to be open as it used a pretrained model trained with external data. Our team attained shared second position in the shared task with the submission based on Naive Bayes. Our results reinforce the idea that deep learning methods are not as competitive in language identification related tasks as they are in many other text classification tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑