arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2010.04532cs.CLcs.LG
收藏

衡量什么重要:谣言立场分类案例

Measuring What Counts: The case of Rumour Stance Classification

University of Sheffield(谢菲尔德大学);Federal University of São Carlos(圣卡洛斯联邦大学)

Carolina Scarton, Diego F. Silva, Kalina Bontcheva

摘要:

立场分类可以成为了解用户是否以及哪些用户相信在线谣言的强大工具。该任务旨在自动预测回复对给定谣言的立场,即支持,否认,质疑或评论。已经提出了许多方法,并在2017年和2019年的RumourEval共享任务中比较了它们的性能。结果表明,这是一个具有挑战性的问题,因为自然发生的谣言立场数据是高度不平衡的。本文特别质疑这些共享任务中使用的评估指标。我们重新评估了提交给两个RumourEval任务的系统,并表明两个广泛采用的指标-准确性和宏观F1-对于谣言立场分类的四类不平衡任务并不稳健,因为它们错误地支持具有高度倾斜准确性的系统。为了克服这个问题,我们提出了新的评价指标的谣言立场检测。这些不仅对不平衡数据具有鲁棒性,而且能够识别两个信息量最大的少数类别(支持和拒绝)的系统得分更高。

英文摘要:

Stance classification can be a powerful tool for understanding whether and which users believe in online rumours. The task aims to automatically predict the stance of replies towards a given rumour, namely support, deny, question, or comment. Numerous methods have been proposed and their performance compared in the RumourEval shared tasks in 2017 and 2019. Results demonstrated that this is a challenging problem since naturally occurring rumour stance data is highly imbalanced. This paper specifically questions the evaluation metrics used in these shared tasks. We re-evaluate the systems submitted to the two RumourEval tasks and show that the two widely adopted metrics -- accuracy and macro-F1 -- are not robust for the four-class imbalanced task of rumour stance classification, as they wrongly favour systems with highly skewed accuracy towards the majority class. To overcome this problem, we propose new evaluation metrics for rumour stance detection. These are not only robust to imbalanced data but also score higher systems that are capable of recognising the two most informative minority classes (support and deny).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑