arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.09345cs.CL
收藏

CrossRE:一个用于关系抽取的跨领域数据集

CrossRE: A Cross-Domain Dataset for Relation Extraction

IT University of Copenhagen(哥本哈根信息技术大学);LMU Munich(慕尼黑大学);Munich Center for Machine Learning(慕尼黑机器学习中心)

Elisa Bassignana, Barbara Plank

摘要:

关系抽取(RE)已引起越来越多的关注,但目前的RE评估仅限于域内评估设置。很少有人知道如何以及一个RE系统票价在具有挑战性的,但现实的分布外的评估设置。为了解决这一差距,我们提出了CrossRE,一个新的,免费提供的跨域基准RE,其中包括六个不同的文本域,并包括多标签注释。另一个创新是,我们发布了在注释过程中收集的元数据,包括解释和困难实例的标志。我们提供了一个经验评估与国家的最先进的模型关系分类。由于元数据使我们能够对最先进的模型进行新的阐述,我们对困难案例的影响进行了全面的分析,并发现了模型和人工注释之间的相关性。总的来说,我们的实证研究突出了跨域RE的困难。我们发布了我们的数据集,以刺激更多的研究在这个方向。

英文摘要:

Relation Extraction (RE) has attracted increasing attention, but current RE evaluation is limited to in-domain evaluation setups. Little is known on how well a RE system fares in challenging, but realistic out-of-distribution evaluation setups. To address this gap, we propose CrossRE, a new, freely-available cross-domain benchmark for RE, which comprises six distinct text domains and includes multi-label annotations. An additional innovation is that we release meta-data collected during annotation, to include explanations and flags of difficult instances. We provide an empirical evaluation with a state-of-the-art model for relation classification. As the meta-data enables us to shed new light on the state-of-the-art model, we provide a comprehensive analysis on the impact of difficult cases and find correlations between model and human annotations. Overall, our empirical investigation highlights the difficulty of cross-domain RE. We release our dataset, to spur more research in this direction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑