arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2203.12184cs.CL
收藏

一种基于理论的机器常识评估基准

A Theoretically Grounded Benchmark for Evaluating Machine Commonsense

Rensselaer Polytechnic Institute(伦斯勒理工学院);University of Southern California(南加州大学);University of California, Irvine(加州大学欧文分校)

Henrique Santos, Ke Shen, Alice M. Mulvehill, Yasaman Razeghi, Deborah L. McGuinness, Mayank Kejriwal

摘要:

对具有常识推理(CSR)能力的机器进行编程是人工智能社区的一个长期挑战。目前的CSR基准使用多项选择(在相对较少的情况下,生成)问题回答实例来评估机器常识。基于transformer的语言表示模型的最新进展表明,在现有的基准上已经取得了相当大的进展。然而,尽管目前存在数十个企业社会责任基准,而且还在不断增加,但并不明显的是,整套常识性能力已经得到了系统的评估。此外,还有人怀疑语言模型是否“适合"基准数据集的训练分区,通过挑选微妙的,但规范上不相关的(至少对于CSR),统计特征来实现测试分区的良好性能。为了解决这些挑战,我们提出了一个基准称为理论接地常识推理(TG-CSR),也是基于歧视性的问答,但问题的目的是评估常识的各个方面,如空间,时间和世界状态。TG-CSR是基于一个子集的常识范畴首先提出了一个可行的理论常识戈登和霍布斯。基准测试也被设计为Few-Shot(将来是zero-shot),只提供了一些训练和验证示例。本报告讨论了基准的结构和构造。初步结果表明,即使是先进的语言表示模型设计的区分CSR问答任务的基准是具有挑战性的。基准访问和排行榜:https://codalab.lisn.upsaclay.fr/competitions/3080基准网站:https://usc-isi-i2.github.io/TGCSR/

英文摘要:

Programming machines with commonsense reasoning (CSR) abilities is a longstanding challenge in the Artificial Intelligence community. Current CSR benchmarks use multiple-choice (and in relatively fewer cases, generative) question-answering instances to evaluate machine commonsense. Recent progress in transformer-based language representation models suggest that considerable progress has been made on existing benchmarks. However, although tens of CSR benchmarks currently exist, and are growing, it is not evident that the full suite of commonsense capabilities have been systematically evaluated. Furthermore, there are doubts about whether language models are 'fitting' to a benchmark dataset's training partition by picking up on subtle, but normatively irrelevant (at least for CSR), statistical features to achieve good performance on the testing partition. To address these challenges, we propose a benchmark called Theoretically-Grounded Commonsense Reasoning (TG-CSR) that is also based on discriminative question answering, but with questions designed to evaluate diverse aspects of commonsense, such as space, time, and world states. TG-CSR is based on a subset of commonsense categories first proposed as a viable theory of commonsense by Gordon and Hobbs. The benchmark is also designed to be few-shot (and in the future, zero-shot), with only a few training and validation examples provided. This report discusses the structure and construction of the benchmark. Preliminary results suggest that the benchmark is challenging even for advanced language representation models designed for discriminative CSR question answering tasks. Benchmark access and leaderboard: https://codalab.lisn.upsaclay.fr/competitions/3080 Benchmark website: https://usc-isi-i2.github.io/TGCSR/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑