机器学习在摩尔多瓦语和罗马尼亚语方言识别中的不合理有效性
The Unreasonable Effectiveness of Machine Learning in Moldavian versus Romanian Dialect Identification
摘要：
由于机器学习模型在摩尔多瓦语与罗马尼亚语方言识别中的准确性似乎很高，并且对这一主题的研究兴趣越来越大，我们对摩尔多瓦语与罗马尼亚语跨方言主题识别（MRC）VarDial 2019评估活动的共享任务进行了后续研究。共享的任务包括两个子任务类型：一个是在摩尔多瓦和罗马尼亚方言之间进行区分，一个是在罗马尼亚语的两种方言之间按主题对文件进行分类。参与者取得了令人印象深刻的分数，例如，摩尔多瓦语与罗马尼亚语方言识别的顶级模型获得了0.895的宏观F1分数。我们通过人工注释者进行了主观评估，结果表明，与机器学习（ML）模型相比，人类的准确率要低得多。因此，目前尚不清楚为什么参与者提出的方法能达到如此高的准确率。我们的目标是理解（i）为什么所提出的方法工作得如此之好（通过可视化区分特征）以及（ii）这些方法在多大程度上可以保持其高准确度，例如，当我们将文本样本缩短为单个句子时，或者当我们在推理时使用推文时。我们工作的第二个目标是使用集成学习提出一个改进的ML模型。我们的实验表明，ML模型可以准确地识别方言，即使在句子级别和不同的领域（新闻文章与推文）。我们还分析了表现最好的模型的最具区别性的特征，为这些模型所做的决定提供了一些解释。有趣的是，我们学习到了我们或我们的人类注释者以前不知道的新方言模式。此外，我们进行的实验表明，机器学习的MRC共享任务的性能可以通过集成的基础上堆叠提高。
英文摘要：
Motivated by the seemingly high accuracy levels of machine learning models in Moldavian versus Romanian dialect identification and the increasing research interest on this topic, we provide a follow-up on the Moldavian versus Romanian Cross-Dialect Topic Identification (MRC) shared task of the VarDial 2019 Evaluation Campaign. The shared task included two sub-task types: one that consisted in discriminating between the Moldavian and Romanian dialects and one that consisted in classifying documents by topic across the two dialects of Romanian. Participants achieved impressive scores, e.g. the top model for Moldavian versus Romanian dialect identification obtained a macro F1 score of 0.895. We conduct a subjective evaluation by human annotators, showing that humans attain much lower accuracy rates compared to machine learning (ML) models. Hence, it remains unclear why the methods proposed by participants attain such high accuracy rates. Our goal is to understand (i) why the proposed methods work so well (by visualizing the discriminative features) and (ii) to what extent these methods can keep their high accuracy levels, e.g. when we shorten the text samples to single sentences or when we use tweets at inference time. A secondary goal of our work is to propose an improved ML model using ensemble learning. Our experiments show that ML models can accurately identify the dialects, even at the sentence level and across different domains (news articles versus tweets). We also analyze the most discriminative features of the best performing models, providing some explanations behind the decisions taken by these models. Interestingly, we learn new dialectal patterns previously unknown to us or to our human annotators. Furthermore, we conduct experiments showing that the machine learning performance on the MRC shared task can be improved through an ensemble based on stacking.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
