HateDay：来自一个代表Twitter上一天情况的全球仇恨言论数据集的见解
HateDay: Insights from a Global Hate Speech Dataset Representative of a Day on Twitter
AI总结：
HateDay构建了首个基于Twitter全天随机抽样的全球仇恨言论数据集，覆盖八种语言和四个英语国家，揭示了学术评估高估现实检测性能的问题，并指出模型需人工监督才能有效审核。
摘要：
为应对全球范围内的网络仇恨言论挑战，先前研究已开发出检测模型来标记社交媒体上的此类内容。然而，由于评估数据集中存在系统性偏差，这些模型在现实世界中的有效性仍不明确，尤其是在不同地理区域之间。我们引入了HateDay，这是第一个代表社交媒体环境的全球仇恨言论数据集，该数据集从2022年9月21日发布的所有推文中随机抽样构建，涵盖八种语言和四个英语国家。利用HateDay，我们揭示了仇恨言论在不同语言和地区之间的流行程度和构成存在显著差异。我们表明，在学术数据集上的评估极大地高估了现实世界的检测性能，而我们发现实际检测性能非常低，尤其是对于非欧洲语言。我们的分析确定了造成这一差距的关键驱动因素，包括模型难以区分仇恨言论和冒犯性言论，以及学术数据集中强调的目标群体与现实环境中最常被针对的群体之间的不匹配。我们认为，糟糕的模型性能使得公开模型不适合用于自动仇恨言论审核，并发现只有在大量人工监督下才能实现较高的审核率。我们的结果强调，需要在能够反映现实世界社交媒体复杂性和多样性的数据上评估检测系统。
英文摘要：
To address the global challenge of online hate speech, prior research has developed detection models to flag such content on social media. However, due to systematic biases in evaluation datasets, the real-world effectiveness of these models remains unclear, particularly across geographies. We introduce HateDay, the first global hate speech dataset representative of social media settings, constructed from a random sample of all tweets posted on September 21, 2022 and covering eight languages and four English-speaking countries. Using HateDay, we uncover substantial variation in the prevalence and composition of hate speech across languages and regions. We show that evaluations on academic datasets greatly overestimate real-world detection performance, which we find is very low, especially for non-European languages. Our analysis identifies key drivers of this gap, including models' difficulty to distinguish hate from offensive speech and a mismatch between the target groups emphasized in academic datasets and those most frequently targeted in real-world settings. We argue that poor model performance makes public models ill-suited for automatic hate speech moderation and find that high moderation rates are only achievable with substantial human oversight. Our results underscore the need to evaluate detection systems on data that reflects the complexity and diversity of real-world social media.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
