arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2007.15066cs.CL
收藏

位置偏向对情绪影响的实验研究CauseExtraction

An Experimental Study of The Effects of Position Bias on Emotion CauseExtraction

VMware(威睿);Information Sciences Institute(信息科学研究所);University of Southern California(南加州大学)

Jiayuan Ding, Mayank Kejriwal

摘要:

情感原因提取(ECE)的目的是在标注情感关键字后,从文档中识别情感原因。已经提出了一些基线来解决这个问题,例如基于规则的方法、基于常识的方法和机器学习方法。然而,我们发现,一个简单的随机选择方法对ECE,不需要观察文本实现类似的性能相比,基线。我们只利用相对于情绪原因的位置信息来实现这一目标。由于位置信息单独没有观察到的文本导致更高的F-措施,因此,我们发现在欧洲经委会单一体裁新浪新闻基准的偏见。进一步的分析表明,情感原因位置的不平衡存在的基准,与大部分的原因子句紧接在中央情感子句。我们从语言学的角度研究了这种偏差,并表明利用位置信息的当前最先进的深度学习模型的高准确率仅在包含这种位置偏差的数据集中是明显的。当引入具有平衡位置分布的数据集时,准确性急剧降低。因此,我们得出结论,正是这个基准测试中的固有偏差导致了ECE中这些深度学习模型的高准确率。我们希望本文中的案例研究既能提供警示性的教训,也能为进一步的研究提供模板,以解释深度学习模型的卓越拟合,而无需检查偏见。

英文摘要:

Emotion Cause Extraction (ECE) aims to identify emotion causes from a document after annotating the emotion keywords. Some baselines have been proposed to address this problem, such as rule-based, commonsense based and machine learning methods. We show, however, that a simple random selection approach toward ECE that does not require observing the text achieves similar performance compared to the baselines. We utilized only position information relative to the emotion cause to accomplish this goal. Since position information alone without observing the text resulted in higher F-measure, we therefore uncovered a bias in the ECE single genre Sina-news benchmark. Further analysis showed that an imbalance of emotional cause location exists in the benchmark, with a majority of cause clauses immediately preceding the central emotion clause. We examine the bias from a linguistic perspective, and show that high accuracy rate of current state-of-art deep learning models that utilize location information is only evident in datasets that contain such position biases. The accuracy drastically reduced when a dataset with balanced location distribution is introduced. We therefore conclude that it is the innate bias in this benchmark that caused high accuracy rate of these deep learning models in ECE. We hope that the case study in this paper presents both a cautionary lesson, as well as a template for further studies, in interpreting the superior fit of deep learning models without checking for bias.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑