arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2106.06132cs.CL
收藏

TellMeWhy:一个用于描述叙述中的Why问题的数据集

TellMeWhy: A Dataset for Answering Why-Questions in Narratives

Stony Brook University(石溪大学);US Naval Academy(美国海军学院);University of Texas, Austin(德克萨斯大学奥斯汀分校)

Yash Kumar Lal, Nathanael Chambers, Raymond Mooney, Niranjan Balasubramanian

摘要:

关于人物为什么要做某些动作的问题是理解和推理叙事的核心。尽管最近在QA方面取得了进展,但尚不清楚现有模型是否有能力回答“为什么”的问题,这些问题可能需要输入叙述之外的常识知识。在这项工作中,我们介绍了TellMeWhy,这是一个新的众包数据集,包含超过30,000个问题和自由形式的答案,这些问题和答案涉及为什么简短叙述中的人物会执行所描述的动作。对于这个数据集的三分之一,答案不存在于叙述中。考虑到自动评估这项任务的局限性,我们还提出了一个系统化的人类评估界面,这个数据集。我们对最先进模型的评估表明,它们在回答这些问题时远远低于人类的表现。他们在回答叙述之外的问题时尤其糟糕,因此为未来的问答和叙述理解研究提供了挑战。

英文摘要:

Answering questions about why characters perform certain actions is central to understanding and reasoning about narratives. Despite recent progress in QA, it is not clear if existing models have the ability to answer "why" questions that may require commonsense knowledge external to the input narrative. In this work, we introduce TellMeWhy, a new crowd-sourced dataset that consists of more than 30k questions and free-form answers concerning why characters in short narratives perform the actions described. For a third of this dataset, the answers are not present within the narrative. Given the limitations of automated evaluation for this task, we also present a systematized human evaluation interface for this dataset. Our evaluation of state-of-the-art models show that they are far below human performance on answering such questions. They are especially worse on questions whose answers are external to the narrative, thus providing a challenge for future QA and narrative understanding research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑