接近人类水平的共识：使用GPT-4增强的标注追踪Incel帖子中暴力言论的演变历程
Close to Human-Level Agreement: Tracing Journeys of Violent Speech in Incel Posts with GPT-4-Enhanced Annotations
AI总结：
本研究评估了GPT模型在incels.is暴力言论检测中的编码质量，发现GPT-4与人工标注高度一致，揭示了暴力语言上升趋势，为大规模内容审核和后续研究提供了实用方法。
摘要：
本研究调查了incels.is上暴力语言的普遍程度。它评估了GPT模型（GPT-3.5和GPT-4）在社会科学内容分析中的表现，重点关注不同提示词和批处理大小对暴力言论检测编码质量的影响。我们从incels.is抓取了超过690万条帖子，并将一个随机样本分为非暴力、显性暴力和隐性暴力内容。两名人工编码员对3,028条帖子进行了标注，我们以此在不同提示词和批处理大小下调整和评估GPT-3.5和GPT-4模型的编码可靠性。表现最佳的GPT-4模型对额外的30,000条帖子进行了标注以供进一步分析。我们的发现表明，incels.is上的暴力言论随时间推移总体呈上升趋势，无论是在社区层面还是个人层面，尤其是在参与度较高的用户中。虽然定向暴力语言有所减少，但非定向暴力语言有所增加，自我伤害内容呈现下降趋势，特别是在用户活动2.5年后。我们发现两名人工编码员之间存在实质性共识（K = .65），而最佳的GPT-4模型与两名人工编码员均取得了良好共识（人类A的K = 0.54，人类B的K = 0.62）。加权和宏F1分数进一步支持了这种一致性。总体而言，本研究提供了大规模准确识别暴力语言的实用方法，可辅助内容审核，并促进对incels.is等社区中暴力表达和激进化因果机制及潜在缓解措施的后续研究。
英文摘要：
This study investigates the prevalence of violent language on incels.is. It evaluates GPT models (GPT-3.5 and GPT-4) for content analysis in social sciences, focusing on the impact of varying prompts and batch sizes on coding quality for the detection of violent speech. We scraped over 6.9M posts from incels.is and categorized a random sample into non-violent, explicitly violent, and implicitly violent content. Two human coders annotated 3,028 posts, which we used to tune and evaluate GPT-3.5 and GPT-4 models across different prompts and batch sizes regarding coding reliability. The best-performing GPT-4 model annotated an additional 30,000 posts for further analysis. Our findings indicate an overall increase in violent speech overtime on incels.is, both at the community and individual level, particularly among more engaged users. While directed violent language decreases, non-directed violent language increases, and self-harm content shows a decline, especially after 2.5 years of user activity. We find substantial agreement between both human coders (K = .65), while the best GPT-4 model yields good agreement with both human coders (K = 0.54 for Human A and K = 0.62 for Human B). Weighted and macro F1 scores further support this alignment. Overall, this research provides practical means for accurately identifying violent language at a large scale that can aid content moderation and facilitate next-step research into the causal mechanism and potential mitigations of violent expression and radicalization in communities like incels.is.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
