This article benchmarked the ability of OpenAI's GPTs and a number of open-source LLMs to perform annotation tasks on political content. We used a novel protest event dataset comprising more than three million digital interactions and created a gold standard that includes ground-truth labels annotated by human coders about toxicity and incivility on social media. We included in our benchmark Google's Perspective algorithm, which, along with GPTs, was employed throughout their respective APIs while the open-source LLMs were deployed locally. The findings show that Perspective API using a laxer threshold, GPT-4o, and Nous Hermes 2 Mixtral outperform other LLM's zero-shot classification annotations. In addition, Nous Hermes 2 and Mistral OpenOrca, with a smaller number of parameters, are able to perform the task with high performance, being attractive options that could offer good trade-offs between performance, implementing costs and computing time. Ancillary findings using experiments setting different temperature levels show that although GPTs tend to show not only excellent computing time but also overall good levels of reliability, only open-source LLMs ensure full reproducibility in the annotation.
Polarization by Default: Auditing Recommendation Bias in LLM-Based Content Curation
University of Zurich, Department of Informatics（苏黎世大学信息学院）；New York University, Department of Sociology（纽约大学社会学系）；University of Oxford, Department of Sociology（牛津大学社会学系）；Duke University, Department of Sociology, Computer Science, Political Science, and Public Policy（杜克大学社会学系、计算机科学、政治学和公共政策系）；University of Amsterdam, Institute for Logic, Language and Computation (ILLC)（阿姆斯特丹大学逻辑、语言与计算研究所（ILLC））
From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language Models
University of Michigan（密歇根大学）；School of Information（信息学院）；Allen Institute for AI（艾伦人工智能研究所）；Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）
