As large language models (LLMs) become more capable, there is growing excitement about the possibility of using LLMs as proxies for humans in real-world tasks where subjective labels are desired, such as in surveys and opinion polling. One widely-cited barrier to the adoption of LLMs as proxies for humans in subjective tasks is their sensitivity to prompt wording - but interestingly, humans also display sensitivities to instruction changes in the form of response biases. We investigate the extent to which LLMs reflect human response biases, if at all. We look to survey design, where human response biases caused by changes in the wordings of "prompts" have been extensively explored in social psychology literature. Drawing from these works, we design a dataset and framework to evaluate whether LLMs exhibit human-like response biases in survey questionnaires. Our comprehensive evaluation of nine models shows that popular open and commercial LLMs generally fail to reflect human-like behavior, particularly in models that have undergone RLHF. Furthermore, even if a model shows a significant change in the same direction as humans, we find that they are sensitive to perturbations that do not elicit significant changes in humans. These results highlight the pitfalls of using LLMs as human proxies, and underscore the need for finer-grained characterizations of model behavior. Our code, dataset, and collected samples are available at https://github.com/lindiatjuatja/BiasMonkey
The Biased Oracle: Assessing LLMs' Understandability and Empathy in Medical Diagnoses
Department of Chemistry and Applied Biosciences, ETH Zurich(应用生物科学系,苏黎世联邦理工学院);Department of Computer Science, ETH Zurich(计算机科学系,苏黎世联邦理工学院);Department of Health Sciences and Technology, ETH Zurich(健康科学与技术系,苏黎世联邦理工学院)
Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages
Georgia Institute of Technology(佐治亚理工学院);Samsung R&D Institute Philippines(三星菲律宾研发院);Sungkyunkwan University(成均馆大学);Tohoku University(东北大学);National University of Singapore(新加坡国立大学);University of Copenhagen(哥本哈根大学);University of Michigan(密歇根大学);Indian Institute of Technology Delhi(印度理工学院德里);Institute of Science Tokyo(东京科学研究院)
Desert Camels and Oil Sheikhs: Arab-Centric Red Teaming of Frontier LLMs
MBZUAI(穆罕默德·本·扎耶德人工智能大学);University of Khartoum(喀土穆大学);Universität des Saarlandes(萨尔大学);University of British Columbia(不列颠哥伦比亚大学);Invertible AI(可逆人工智能公司);Vector Institute(向量研究所)
A word association network methodology for evaluating implicit biases in LLMs compared to humans
University of Pisa, Department of Computer Science(比萨大学计算机科学系);National Research Council of Italy, Institute of Information Science and Technologies(意大利国家研究委员会信息科学与技术研究所);University of Trento, Department of Psychology and Cognitive Science(特伦托大学心理学与认知科学系)
Colombian Waitresses y Jueces canadienses: Gender and Country Biases in Occupation Recommendations from LLMs
Erasmus Mundus Master in Language and Communication Technologies(埃拉斯谟 Mundus 语言与沟通技术硕士项目);HiTZ Center - Ixa, University of the Basque Country (UPV/EHU)(巴斯克国家大学(UPV/EHU)HiTZ中心 - Ixa);Cohere(Cohere公司);Saarland University(萨尔兰州大学)
LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs
National University of Singapore(新加坡国立大学);VinAI Research(VinAI研究院);Institute of High Performance Computing (IHPC), A*STAR(新加坡科技研究局高性能计算研究所);Salesforce Research(Salesforce研究院);Nanyang Technological University(南洋理工大学)
Large Language Models are biased to overestimate profoundness
Pontificia Universidad Católica de Chile(智利天主教大学);Centro Nacional de Inteligencia Artificial (CENIA)(国家人工智能中心);Brown University(布朗大学);Leiden University(莱顿大学)
Shaping Shared Languages: Human and Large Language Models' Inductive Biases in Emergent Communication
Leiden Institute of Advanced Computer Science, Leiden University, Netherlands(莱顿高级计算机科学研究所,莱顿大学,荷兰);School of Computing, University of Kent, United Kingdom(计算学院,肯特大学,英国);Cognitive Psychology Unit, Institute of Psychology, Leiden, the Netherlands(认知心理学单元,心理学研究所,莱顿,荷兰)
The role of System 1 and System 2 semantic memory structure in human and LLM biases
Department of Computer Science, University of Pisa, Italy(意大利比萨大学计算机科学系);Institute of Information Science and Technologies - National Research Council, Italy(意大利信息科学与技术研究所-国家研究理事会);CogNosco Lab, Department of Psychology and Cognitive Science, University of Trento, Italy(意大利特伦托大学心理学与认知科学系CogNosco实验室)
Counterfactual Debating with Preset Stances for Hallucination Elimination of LLMs
University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学);Electronic Science Research Institute of China Electronics(中国电子科技集团电子科学研究院)
Mitigating Biases of Large Language Models in Stance Detection with Counterfactual Augmented Calibration
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Peng Cheng Laboratory(鹏城实验室);The Chinese University of Hong Kong(香港中文大学);King’s College London(伦敦国王学院);The 30th Research Institute of China Electronics Technology Group Corporation(中国电子科技集团公司第三十研究所)