利用社区和作者上下文解释基于文本的欺骗检测模型的性能和偏差
Leveraging Community and Author Context to Explain the Performance and Bias of Text-Based Deception Detection Models
摘要:
在网络社区中分享的欺骗性新闻帖子可以用NLP模型检测出来,最近的许多研究都集中在这种模型的开发上。在这项工作中,我们使用在线社区和作者的特征-内容发布的方式和位置-来解释神经网络欺骗检测模型的性能,并识别受模型准确性或失败影响不成比例的子人群。我们会检查谁在发布内容,以及内容发布到哪里。我们发现,虽然作者特征是更好的预测欺骗性内容比社区特征,这两个特征与模型性能密切相关。传统的性能指标,如F1分数可能无法捕捉孤立的子群体,如特定的作者,模型性能差,因此,更细致的评估欺骗检测模型是至关重要的。
英文摘要:
Deceptive news posts shared in online communities can be detected with NLP models, and much recent research has focused on the development of such models. In this work, we use characteristics of online communities and authors -- the context of how and where content is posted -- to explain the performance of a neural network deception detection model and identify sub-populations who are disproportionately affected by model accuracy or failure. We examine who is posting the content, and where the content is posted to. We find that while author characteristics are better predictors of deceptive content than community characteristics, both characteristics are strongly correlated with model performance. Traditional performance metrics such as F1 score may fail to capture poor model performance on isolated sub-populations such as specific authors, and as such, more nuanced evaluation of deception detection models is critical.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。