Large language models (LLMs) are rapidly being integrated into computational social science research, yet their blackboxed training and designed stochastic elements in inference pose unique challenges for scientific inquiry. This article argues that applying LLMs to social scientific tasks requires explicit assessment of uncertainty -- an expectation long established in both quantitative methodology in the social sciences and machine learning. We introduce a unified framework for evaluating LLM uncertainty based on Hill numbers, a family of diversity measures. By transforming existing uncertainty quantification (UQ) metrics into Hill numbers, the framework provides a common and intuitive scale for interpreting variation in LLM outputs while accommodating different notions of semantic similarity and different sensitivities to output distributions. We show how it might help the application of LLMs in social sciences through four empirical applications.
Intelligent Computing Social Modeling and Methodological Innovations in Political Science in the Era of Large Language Models
Shanghai Academy of Social Sciences(上海社会科学院);Nanjing University(南京大学);Shanghai Jiao Tong University(上海交通大学);Shanghai Institutes for International Studies(上海国际问题研究院)
Knowing You Is Everything: LLM Agents Achieve Near-Perfect Profile-Consistent Reaction Prediction in Social Media Simulation
Institute for Artificial Intelligence Research and Development of Serbia(塞尔维亚人工智能研发研究所);University of Belgrade(贝尔格莱德大学);Institute for Philosophy and Social Theory(哲学与社会理论研究所);Complexity Science Hub(维也纳复杂性科学中心);University of Kragujevac(克拉古耶瓦茨大学);Faculty of Economics(经济学院);University of Vienna(维也纳大学);Faculty of Social Sciences(社会科学学院);Faculty of Engineering(工程学院);University of Novi Sad(诺维萨德大学);Faculty of Philosophy(哲学院);Nanyang Technological University(南洋理工大学);School of Social Sciences(社会科学学院)
Bridging Technology and Humanities: Evaluating the Impact of Large Language Models on Social Sciences Research with DeepSeek-R1
School of Physics and Information Technology, Shaanxi Normal University(陕西师范大学物理与信息技术学院);School of Foreign Languages, Northwest University(西北大学外国语学院);School of Computing, University of Georgia(佐治亚大学计算学院)
Prompt Selection Matters: Enhancing Text Annotations for Social Sciences with Large Language Models
Université Paris 1 Panthéon-Sorbonne(巴黎第一大学先贤祠-索邦大学);Université Paris-Saclay(巴黎萨克雷大学);Institut Jean Nicod(让·尼科德研究所);Ecole Normale Supérieure(巴黎高等师范学院);PSL-EHESS-CNRS(巴黎文理研究大学-法国高等社会科学研究院-法国国家科学研究中心)
LLM Agents as Social Scientists: A Human-AI Collaborative Platform for Social Science Automation
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Beijing Key Laboratory of Big Data Management and Analysis Methods(大数据管理与分析方法北京市重点实验室)
MASS: Deep Research for Social Sciences with Memory-Augmented Social Simulation
The International Joint Institute of Tianjin University, Fuzhou, Tianjin University, China(天津大学福州国际联合学院);TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China(天津大学计算机科学与技术学院TJUNLP实验室)
Artificially intelligent agents in the social and behavioral sciences: A history and outlook
Department of Computer Science, Aalto University(阿尔托大学计算机科学系);Center for Computational Social Science, Kobe University(大阪大学计算社会科学中心);Department of Methodology, London School of Economics and Political Science(伦敦政治经济学院方法论系)
A High-Scale Assessment of Social Media and Mainstream Media in Scientific Communication
Mendoza College of Business, University of Notre Dame(圣母门多萨商学院,圣母大学);New York University Shanghai(纽约大学上海分校);New York University(纽约大学);Kellogg School of Management, Northwestern University(西北大学凯洛格管理学院);National Bureau of Economic Research(美国国家经济研究局)
Artificial intelligence adoption in the physical sciences, natural sciences, life sciences, social sciences and the arts and humanities: A bibliometric analysis of research publications from 1960-2021
From Symbols to Embeddings: A Tale of Two Representations in Computational Social Science
计算社会科学(Computational Social Science,CSS)是近年来兴起的一门新兴学科,旨在利用计算方法解决社会科学问题。CSS的研究是数据驱动的,并大大受益于在线用户生成的内容和社交网络的可用性,其中包含富文本和网络数据的调查。然而,这些大规模和多模态的数据也给研究人员提出了一个巨大的挑战:如何有效地表示数据,以挖掘我们想要的意义在CSS?为了探索答案,我们对CSS中的文本和网络数据表示进行了全面的回顾。具体来说,我们总结现有的表示为两种方案,即基于符号和基于嵌入的表示,并介绍了一系列的典型方法,为每一个方案。之后,我们提出了上述表示的应用程序的基础上调查的400多篇研究文章,从6个顶级场馆涉及CSS。从这些应用的统计数据中,我们发现了每种表示的强度,并发现了基于嵌入的表示在过去十年中出现并获得越来越多关注的趋势。最后,我们讨论了几个关键的挑战和开放的问题,为未来的发展方向。本调查旨在为CSS研究人员提供更深入的理解和更明智的数据表示应用。