arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.07566cs.CLcs.CV
收藏

NLP中与基准质量相关的参数综述

A Survey of Parameters Associated with the Quality of Benchmarks in NLP

Arizona State University(亚利桑那州立大学)

Swaroop Mishra, Anjana Arunkumar, Chris Bryan, Chitta Baral

摘要:

我们已经建立了几个基准,并投入大量资源来跟踪我们在NLP方面的进展。数以千计针对这些基准发表的论文已经在排行榜上名列前茅,模型往往超过人类的表现。然而,最近的研究表明,模型只是通过对虚假偏差的过度拟合而战胜了几个流行的基准,而没有真正学习所需的任务。尽管有这一发现,基准测试在试图解决偏差的同时,仍然依赖于变通方法,由于丢弃低质量数据,这些变通方法没有充分利用在基准测试创建中投入的资源,并且覆盖有限的偏差。这些问题的一个可能解决办法-量化质量的指标-仍然没有得到充分探讨。受电力、食品和水等多个领域成功的质量指标的启发,我们通过识别某些语言属性来迈出度量的第一步,这些语言属性可以代表导致基准偏差的各种可能的相互作用。我们寻找与偏差相关的参数,这些参数可能有助于我们实现指标。我们调查现有的作品,并确定参数捕捉各种属性的偏见,其起源,类型和影响的性能,泛化和鲁棒性。我们的分析跨越了数据集和从NLI到汇总的任务层次,确保我们的参数是通用的,不会过度拟合特定的任务或数据集。我们还在这个过程中制定了一些参数。

英文摘要:

Several benchmarks have been built with heavy investment in resources to track our progress in NLP. Thousands of papers published in response to those benchmarks have competed to top leaderboards, with models often surpassing human performance. However, recent studies have shown that models triumph over several popular benchmarks just by overfitting on spurious biases, without truly learning the desired task. Despite this finding, benchmarking, while trying to tackle bias, still relies on workarounds, which do not fully utilize the resources invested in benchmark creation, due to the discarding of low quality data, and cover limited sets of bias. A potential solution to these issues -- a metric quantifying quality -- remains underexplored. Inspired by successful quality indices in several domains such as power, food, and water, we take the first step towards a metric by identifying certain language properties that can represent various possible interactions leading to biases in a benchmark. We look for bias related parameters which can potentially help pave our way towards the metric. We survey existing works and identify parameters capturing various properties of bias, their origins, types and impact on performance, generalization, and robustness. Our analysis spans over datasets and a hierarchy of tasks ranging from NLI to Summarization, ensuring that our parameters are generic and are not overfitted towards a specific task or dataset. We also develop certain parameters in this process.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑