This paper investigates bias in GLLM annotations by conceptually replicating manual annotations of Boukes (2024). Using various GLLMs (Llama3.1:8b, Llama3.3:70b, GPT4o, Qwen2.5:72b) in combination with five different prompts for five concepts (political content, interactivity, rationality, incivility, and ideology). We find GLLMs perform adequate in terms of F1 scores, but differ from manual annotations in terms of prevalence, yield substantively different downstream results, and display systematic bias in that they overlap more with each other than with manual annotations. Differences in F1 scores fail to account for the degree of bias.
LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs
National University of Singapore(新加坡国立大学);VinAI Research(VinAI研究院);Institute of High Performance Computing (IHPC), A*STAR(新加坡科技研究局高性能计算研究所);Salesforce Research(Salesforce研究院);Nanyang Technological University(南洋理工大学)
Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought
Speechmatics, Apollo Research(Speechmatics与Apollo研究机构);University of Oxford(牛津大学);NYU, Anthropic(纽约大学与Anthropic);NYU(纽约大学);Anthropic, NYU(Anthropic与纽约大学)
Prompt Selection Matters: Enhancing Text Annotations for Social Sciences with Large Language Models
Université Paris 1 Panthéon-Sorbonne(巴黎第一大学先贤祠-索邦大学);Université Paris-Saclay(巴黎萨克雷大学);Institut Jean Nicod(让·尼科德研究所);Ecole Normale Supérieure(巴黎高等师范学院);PSL-EHESS-CNRS(巴黎文理研究大学-法国高等社会科学研究院-法国国家科学研究中心)
Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks
Information Sciences Institute, University of Southern California(南加州大学信息科学研究所);School of Communication Research, University of Amsterdam(阿姆斯特丹大学传播研究学院);Computer Science Department, University of Turin(都灵大学计算机科学系)
Sent2Span: Span Detection for PICO Extraction in the Biomedical Text without Span Annotations
The University of Sydney(悉尼大学);Northeastern University(东北大学);IHPC, A*STAR(新加坡科技研究局高性能计算研究所);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Harvard Medical School(哈佛医学院)
Greedy Gradient Ensemble for Robust Visual Question Answering
Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS(中国科学院计算技术研究所智能信息处理重点实验室);University of Chinese Academy of Sciences(中国科学院大学);Kingsoft Cloud(金山云);Peng Cheng Laboratory(鹏城实验室);Huawei Technologies(华为技术有限公司)
Exploring Imbalanced Annotations for Effective In-Context Learning
Department of Statistics and Data Science, Southern University of Science and Technology(南方科技大学统计与数据科学系);School of Economics and Finance, Xi’an Jiaotong University(西安交通大学经济金融学院);School of Mathematics and Statistics, Xi’an Jiaotong University(西安交通大学数学与统计学学院)