LLM-based Human Simulations Have Not Yet Been Reliable
National University of Singapore(新加坡国立大学);The Hong Kong University of Science and Technology(香港理工大学);University of Chinese Academy of Sciences(中国科学院大学)
Qian Wang, Jiaying Wu, Zichen Jiang, Zhenheng Tang, Bingqiao Luo, Nuo Chen, Wei Chen, Huacan Wang, Bingsheng He
Large Language Models (LLMs) are increasingly employed for simulating human behaviors across diverse domains. However, our position is that current LLM-based human simulations remain insufficiently reliable, as evidenced by significant discrepancies between their outcomes and authentic human actions. Our investigation begins with a systematic review of LLM-based human simulations in social, economic, policy, and psychological contexts, identifying their common frameworks, recent advances, and persistent limitations. This review reveals that such discrepancies primarily stem from inherent limitations of LLMs and flaws in simulation design, both of which are examined in detail. Building on these insights, we propose a systematic solution framework that emphasizes enriching data foundations, advancing LLM capabilities, and ensuring robust simulation design to enhance reliability. Finally, we introduce a structured algorithm that operationalizes the proposed framework, aiming to guide credible and human-aligned LLM-based simulations. To facilitate further research, we provide a curated list of related literature and resources at https://github.com/Persdre/awesome-llm-human-simulation.
LLMs Capture Emotion Labels, Not Emotion Uncertainty: Distributional Analysis and Calibration of Human-LLM Judgment Gaps
Faculty of Business and Commerce(商科学院);Faculty of Business Data Science(商务数据科学学院);RIKEN Center for Advanced Intelligence Project(先进智能项目研究所);Faculty of Data Science(数据科学学院);Japan Safety Society Research Center(日本安全协会研究中心)
Shaping Shared Languages: Human and Large Language Models' Inductive Biases in Emergent Communication
Leiden Institute of Advanced Computer Science, Leiden University, Netherlands(莱顿高级计算机科学研究所,莱顿大学,荷兰);School of Computing, University of Kent, United Kingdom(计算学院,肯特大学,英国);Cognitive Psychology Unit, Institute of Psychology, Leiden, the Netherlands(认知心理学单元,心理学研究所,莱顿,荷兰)
To model human linguistic prediction, make LLMs less superhuman
Division of Linguistics and Multilingual Studies, Nanyang Technological University, Singapore(南洋理工大学语言学与多语言研究系,新加坡);Department of Linguistics, New York University, New York, USA(纽约大学语言学系,纽约,美国);Center for Data Science, New York University, New York, USA(纽约大学数据科学中心,纽约,美国)
The role of System 1 and System 2 semantic memory structure in human and LLM biases
Department of Computer Science, University of Pisa, Italy(意大利比萨大学计算机科学系);Institute of Information Science and Technologies - National Research Council, Italy(意大利信息科学与技术研究所-国家研究理事会);CogNosco Lab, Department of Psychology and Cognitive Science, University of Trento, Italy(意大利特伦托大学心理学与认知科学系CogNosco实验室)
Presupposition and Reasoning in Conditionals: A Theory-Based Study of Humans and LLMs
Department of Cognitive Science, Carleton University(卡尔顿大学认知科学系);School of Computer Science, McGill University(麦吉尔大学计算机科学学院);Mila – Quebec AI Institute(魁北克人工智能研究所)
Quantifier Scope Interpretation in Language Learners and LLMs
Department of English, Purdue University(英语系,普渡大学);Department of Linguistics, Purdue University(语言学系,普渡大学);School of Languages and Cultures, Purdue University(语言与文化学院,普渡大学)
Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty
DIPF | Leibniz Institute for Research and Information in Education(DIPF | 莱布尼茨教育与研究信息研究所);Centre for International Student Assessment (ZIB)(国际学生评估中心(ZIB));Faculty of Computer Science, Goethe University Frankfurt(法兰克福大学计算机科学学院);Chemnitz University of Technology(开姆尼茨工业大学)
How Deep is Love in LLMs' Hearts? Exploring Semantic Size in Human-like Cognition
Shanghai Jiao Tong University(上海交通大学);Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3(上海Web3可信数据流通与治理重点实验室);Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering(上海市教委智能交互与认知工程重点实验室)
School of Computer Science and Engineering, Northeastern University, Shenyang, China(计算机科学与工程学院,东北大学,沈阳,中国);NiuTrans Research(NiuTrans研究);Kunming University of Science and Technology(昆明理工大学)
A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI
MaiNLP, Center for Information and Language Processing, LMU Munich, Germany(MaiNLP,信息与语言处理中心,慕尼黑大学,德国);Munich Center for Machine Learning (MCML), Munich, Germany(慕尼黑机器学习中心(MCML),慕尼黑,德国);Language Technology Lab, University of Cambridge, United Kingdom(语言技术实验室,剑桥大学,英国)
Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge
National Institute of Informatics (NII)(国立信息学研究所);University of Amsterdam(阿姆斯特丹大学);University of Tokyo(东京大学);Hitotsubashi University(一桥大学);Centrum Wiskunde & Informatica (CWI)(荷兰数学与计算机科学研究中心);Utrecht University(乌得勒支大学)