经验与预测:一种新型试金石的难度度量
Experience and Prediction: A Metric of Hardness for a Novel Litmus Test
AI总结:
本文提出基于机器学习的系统,用于快速准确评估Winograd模式对人类感知的难度,并展示大规模实验结果,以支持未来应用。
摘要:
在过去十年中,Winograd模式挑战(WSC)作为一种新型试金石,已成为研究界的核心关注点。因此,WSC激发了研究兴趣,因为它可以被视为理解人类行为的手段。在这方面,新技术的发展使得Winograd模式能够在各个领域得到应用,例如设计新型验证码(CAPTCHA)。文献中建立的人类成年人在WSC上表现基线的研究表明,并非所有模式都是相同的,这意味着它们可能根据人类感知的难度进行分类。在这方面,这种难度度量可以用于未来的挑战或WSC验证码服务,以区分不同的Winograd模式。我们最近的工作表明,这可以通过设计一个能够输出Winograd模式难度指数的自动化系统来实现,尽管在可应用的模式数量上存在局限性。本文在先前研究的基础上,提出了一种基于机器学习(ML)的新系统,能够比任何先前使用的方法更快、更准确地输出任何Winograd模式的难度。我们开发的系统采用两种不同的方法,即随机森林和深度学习(基于LSTM),已准备好作为任何其他旨在根据人类感知难度区分Winograd模式的系统的扩展。同时,连同我们开发的系统,我们扩展了先前的工作,展示了一项大规模实验的结果,该实验显示了人类表现在不同Winograd模式之间的差异。
英文摘要:
In the last decade, the Winograd Schema Challenge (WSC) has become a central aspect of the research community as a novel litmus test. Consequently, the WSC has spurred research interest because it can be seen as the means to understand human behavior. In this regard, the development of new techniques has made possible the usage of Winograd schemas in various fields, such as the design of novel forms of CAPTCHAs. Work from the literature that established a baseline for human adult performance on the WSC has shown that not all schemas are the same, meaning that they could potentially be categorized according to their perceived hardness for humans. In this regard, this \textit{hardness-metric} could be used in future challenges or in the WSC CAPTCHA service to differentiate between Winograd schemas. Recent work of ours has shown that this could be achieved via the design of an automated system that is able to output the hardness-indexes of Winograd schemas, albeit with limitations regarding the number of schemas it could be applied on. This paper adds to previous research by presenting a new system that is based on Machine Learning (ML), able to output the hardness of any Winograd schema faster and more accurately than any other previously used method. Our developed system, which works within two different approaches, namely the random forest and deep learning (LSTM-based), is ready to be used as an extension of any other system that aims to differentiate between Winograd schemas, according to their perceived hardness for humans. At the same time, along with our developed system we extend previous work by presenting the results of a large-scale experiment that shows how human performance varies across Winograd schemas.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。