In an often-cited 2019 paper on the use of machine learning in political research, Anastasopoulos & Whitford (A&W) propose a text classification method for tweets related to organizational reputation. The aim of their paper was to provide a 'guide to practice' for public administration scholars and practitioners on the use of machine learning. In the current paper we follow up on that work with a replication of A&W's experiments and additional analyses on model stability and the effects of preprocessing, both in relation to the small data size. We show that (1) the small data causes the classification model to be highly sensitive to variations in the random train-test split, and that (2) the applied preprocessing causes the data to be extremely sparse, with the majority of items in the data having at most two non-zero lexical features. With additional experiments in which we vary the steps of the preprocessing pipeline, we show that the small data size keeps causing problems, irrespective of the preprocessing choices. Based on our findings, we argue that A&W's conclusions regarding the automated classification of organizational reputation tweets -- either substantive or methodological -- can not be maintained and require a larger data set for training and more careful validation.
High Risk of Political Bias in Black Box Emotion Inference Models
Institute of Psychology, Polish Academy of Sciences(波兰科学院心理学研究所);Stowarzyszenie Otwartej Nauki(开放科学协会);Institute of Philosophy and Sociology, Polish Academy of Sciences(波兰科学院哲学与社会学研究所)
Using Large Language Models to Create AI Personas for Replication, Generalization and Prediction of Media Effects: An Empirical Test of 133 Published Experimental Research Findings
100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Models
MiroMind(米罗Mind);Fudan University(复旦大学);National University of Singapore(新加坡国立大学);Singapore University of Technology and Design(新加坡科技设计大学);Nanyang Technological University(南洋理工大学)
Against Political Polarization: A Unified Framework for Tracing Evolving Political Ideologies on Social Media
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));University of Science and Technology of China(中国科学技术大学);The Hong Kong University of Science and Technology(香港科技大学)