arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2105.01192cs.CL
收藏

基于规则的启发式算法在解决俄语SuperGLUE任务中的非合理有效性

Unreasonable Effectiveness of Rule-Based Heuristics in Solving Russian SuperGLUE Tasks

HSE University(高等经济大学);Sberbank(俄罗斯联邦储蓄银行);University of Oslo(奥斯陆大学)

Tatyana Iazykova, Denis Kapelyushnik, Olga Bystrova, Andrey Kutuzov

摘要:

像SuperGLUE这样的排行榜被视为积极发展NLP的重要激励因素,因为它们为现代语言模型的公平比较提供了标准基准。他们已经推动世界上最好的工程团队以及他们的资源来合作和解决一系列通用语言理解的任务。他们的表现分数通常被认为接近甚至高于人类的表现。这些结果鼓励更彻底地分析基准数据集是否具有任何基于机器学习的语言模型可以利用的统计线索。对于英语数据集,结果表明它们通常包含注释工件。这允许用非常简单的规则来解决某些任务,并实现有竞争力的排名。本文对Russian SuperGLUE(RSG)进行了类似的分析,RSG是最近发布的俄语自然语言理解基准集和排行榜。我们表明,它的测试数据集是脆弱的浅层次分析。通常,基于简单规则的方法优于或接近臭名昭著的预训练语言模型(如GPT-3或BERT)的结果。很可能(作为最简单的解释),RSG排行榜中SOTA模型性能的很大一部分是由于利用了这些浅层启发式方法,并且与真正的语言理解没有任何共同点。我们就如何改进这些数据集提供了一系列建议,使RSG排行榜更能代表俄罗斯NLU的实际进展。

英文摘要:

Leader-boards like SuperGLUE are seen as important incentives for active development of NLP, since they provide standard benchmarks for fair comparison of modern language models. They have driven the world's best engineering teams as well as their resources to collaborate and solve a set of tasks for general language understanding. Their performance scores are often claimed to be close to or even higher than the human performance. These results encouraged more thorough analysis of whether the benchmark datasets featured any statistical cues that machine learning based language models can exploit. For English datasets, it was shown that they often contain annotation artifacts. This allows solving certain tasks with very simple rules and achieving competitive rankings. In this paper, a similar analysis was done for the Russian SuperGLUE (RSG), a recently published benchmark set and leader-board for Russian natural language understanding. We show that its test datasets are vulnerable to shallow heuristics. Often approaches based on simple rules outperform or come close to the results of the notorious pre-trained language models like GPT-3 or BERT. It is likely (as the simplest explanation) that a significant part of the SOTA models performance in the RSG leader-board is due to exploiting these shallow heuristics and that has nothing in common with real language understanding. We provide a set of recommendations on how to improve these datasets, making the RSG leader-board even more representative of the real progress in Russian NLU.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑