arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2105.12936cs.CL
收藏

事件QA的语料库水平评估:涵盖2002年古吉拉特邦暴力事件的IndiaPoliceEvents语料库

Corpus-Level Evaluation for Event QA: The IndiaPoliceEvents Corpus Covering the 2002 Gujarat Violence

Massachusetts Institute of Technology(麻省理工学院);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校)

Andrew Halterman, Katherine A. Keith, Sheikh Muhammad Sarwar, Brendan O'Connor

摘要:

社会科学应用中的自动事件提取通常需要语料库级别的评估:例如,跨元数据聚合文本预测和无偏的召回估计。我们结合语料库水平的评估要求与现实世界,社会科学的设置,并介绍了IndiaPoliceEvents语料库-所有21,391句,从1,257英语印度时报的文章在古吉拉特邦的事件在2002年3月。我们训练有素的注释员阅读并标记每一份文档中提到的警察活动事件,从而进行公正的回忆评估。与其他具有结构化事件表示的数据集相比,我们通过提出自然问题来收集注释,并评估三种不同任务的现成模型:句子分类,文档排序和目标事件的时间聚合。我们目前的基线结果从zero-shot BERT为基础的模型微调自然语言推理和段落检索任务。我们新颖的语料库级评估和注释方法可以指导未来类似的社会科学资源的创建。

英文摘要:

Automated event extraction in social science applications often requires corpus-level evaluations: for example, aggregating text predictions across metadata and unbiased estimates of recall. We combine corpus-level evaluation requirements with a real-world, social science setting and introduce the IndiaPoliceEvents corpus--all 21,391 sentences from 1,257 English-language Times of India articles about events in the state of Gujarat during March 2002. Our trained annotators read and label every document for mentions of police activity events, allowing for unbiased recall evaluations. In contrast to other datasets with structured event representations, we gather annotations by posing natural questions, and evaluate off-the-shelf models for three different tasks: sentence classification, document ranking, and temporal aggregation of target events. We present baseline results from zero-shot BERT-based models fine-tuned on natural language inference and passage retrieval tasks. Our novel corpus-level evaluations and annotation approach can guide creation of similar social-science-oriented resources in the future.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑