arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2008.03964cs.CLcs.CVcs.LGcs.SYeess.SY
收藏

DQI:基准评估指南

DQI: A Guide to Benchmark Evaluation

Arizona State University(亚利桑那州立大学)

Swaroop Mishra, Anjana Arunkumar, Bhavdeep Sachdeva, Chris Bryan, Chitta Baral

摘要:

“最先进的”模型A在基准B中超过人类,但在类似的基准C、D和E中失败。B有什么是其他基准没有的?最近的研究提供了答案:虚假的偏见。然而,开发A来解决基准测试B到E并不能保证它能解决未来的基准测试。为了实现一个“真正学习”底层任务的模型,我们需要量化连续基准之间的差异,而不是现有的二进制和黑盒方法。我们提出了一种新的方法来解决这个未充分探索的任务,量化基准的质量,推出一个数据质量指标:DQI。

英文摘要:

A `state of the art' model A surpasses humans in a benchmark B, but fails on similar benchmarks C, D, and E. What does B have that the other benchmarks do not? Recent research provides the answer: spurious bias. However, developing A to solve benchmarks B through E does not guarantee that it will solve future benchmarks. To progress towards a model that `truly learns' an underlying task, we need to quantify the differences between successive benchmarks, as opposed to existing binary and black-box approaches. We propose a novel approach to solve this underexplored task of quantifying benchmark quality by debuting a data quality metric: DQI.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑