arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.12911cs.CL
收藏

政治研究中的小数据问题:一项关键的复制研究

Small data problems in political research: a critical replication study

Institute of Public Administration(公共管理研究所);Leiden University(莱顿大学)

Hugo de Vos, Suzan Verberne

摘要:

在2019年一篇经常被引用的关于机器学习在政治研究中的应用的论文中,Anastasopoulos和Whitford(A&W)提出了一种与组织声誉相关的推文的文本分类方法。他们的论文旨在为公共行政学者和从业人员提供一份关于使用机器学习的“实践指南”。在本文中,我们继续这项工作,复制了A&W的实验,并对模型稳定性和预处理的影响进行了额外的分析,两者都与小数据量有关。我们发现:(1)小数据导致分类模型对随机训练-测试分裂的变化高度敏感,(2)应用的预处理导致数据非常稀疏,数据中的大多数项目最多有两个非零词汇特征。通过额外的实验,我们改变了预处理管道的步骤,我们表明,无论预处理选择如何,小数据大小都会导致问题。根据我们的研究结果,我们认为A&W关于组织声誉推文的自动分类的结论-无论是实质性的还是方法性的-都无法维持,需要更大的数据集进行训练和更仔细的验证。

英文摘要:

In an often-cited 2019 paper on the use of machine learning in political research, Anastasopoulos & Whitford (A&W) propose a text classification method for tweets related to organizational reputation. The aim of their paper was to provide a 'guide to practice' for public administration scholars and practitioners on the use of machine learning. In the current paper we follow up on that work with a replication of A&W's experiments and additional analyses on model stability and the effects of preprocessing, both in relation to the small data size. We show that (1) the small data causes the classification model to be highly sensitive to variations in the random train-test split, and that (2) the applied preprocessing causes the data to be extremely sparse, with the majority of items in the data having at most two non-zero lexical features. With additional experiments in which we vary the steps of the preprocessing pipeline, we show that the small data size keeps causing problems, irrespective of the preprocessing choices. Based on our findings, we argue that A&W's conclusions regarding the automated classification of organizational reputation tweets -- either substantive or methodological -- can not be maintained and require a larger data set for training and more careful validation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑