I report here a comprehensive analysis about the political preferences embedded in Large Language Models (LLMs). Namely, I administer 11 political orientation tests, designed to identify the political preferences of the test taker, to 24 state-of-the-art conversational LLMs, both closed and open source. When probed with questions/statements with political connotations, most conversational LLMs tend to generate responses that are diagnosed by most political test instruments as manifesting preferences for left-of-center viewpoints. This does not appear to be the case for five additional base (i.e. foundation) models upon which LLMs optimized for conversation with humans are built. However, the weak performance of the base models at coherently answering the tests' questions makes this subset of results inconclusive. Finally, I demonstrate that LLMs can be steered towards specific locations in the political spectrum through Supervised Fine-Tuning (SFT) with only modest amounts of politically aligned data, suggesting SFT's potential to embed political orientation in LLMs. With LLMs beginning to partially displace traditional information sources like search engines and Wikipedia, the societal implications of political biases embedded in LLMs are substantial.
Polar: A Benchmark for Evaluating Political Bias in LLMs
Graduate School of Data Science, Seoul National University(首尔大学数据科学研究生院);Dept. of Computer Science and Engineering, Seoul National University(首尔大学计算机科学与工程系)
How Identity and Opinion Shape Political Sycophancy in LLMs
National Yang Ming Chiao Tung University(国立阳明交通大学);Academia Sinica(中央研究院);Research Center for Information Technology Innovation, Academia Sinica(中央研究院资讯科技创新研究中心)
Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMs
Institute for Natural Language Processing, University of Stuttgart(斯图加特大学自然语言处理研究所);Faculty of Electrical Engineering and Computing, University of Zagreb(萨格勒布大学电气与计算机学院);University of Manchester(曼彻斯特大学)
Better Aligned with Survey Respondents or Training Data? Unveiling Political Leanings of LLMs on U.S. Supreme Court Cases
Technical University of Munich(慕尼黑技术大学);Allen Institute for AI(艾伦人工智能研究所);University of Washington(华盛顿大学);LMU Munich & Munich Center for Machine Learning (MCML)(慕尼黑大学及慕尼黑机器学习中心)
Political-LLM: Large Language Models in Political Science
Florida State University(佛罗里达州立大学);University of Southern California(南加利福尼亚大学);Louisiana State University(路易斯安那州立大学);University of Virginia(弗吉尼亚大学);University of Pennsylvania(宾夕法尼亚大学);New York University(纽约大学);Oxford University(牛津大学);Lehigh University(理海大学);Rice University(莱斯大学);Northwestern University(西北大学);NVIDIA(英伟达公司)
PolBiX: Detecting LLMs' Political Bias in Fact-Checking through X-phemisms
Quality & Usability Lab, Technische Universität Berlin(柏林技术大学质量与可用性实验室);German Research Center for Artificial Intelligence (DFKI), Berlin(德国人工智能研究中心(DFKI));Department of Media and Communication, Ludwig-Maximilians-Universität München(慕尼黑路德维希-马克西米利安大学媒体与传播系)
Laboratoire Interdisciplinaire Carnot de Bourgogne, Université de Bourgogne(勃艮第大学卡诺跨学科实验室);National University of Computer and Emerging Sciences (NUCES)(国家计算机与新兴科学大学)
Evaluating Large Language Models Against Human Annotators in Latent Content Analysis: Sentiment, Political Leaning, Emotional Intensity, and Sarcasm
Institute for Artificial Intelligence Research and Development of Serbia(塞尔维亚人工智能研究与发展研究所);University of Belgrade(贝尔格莱德大学);Institute for Philosophy and Social Theory(哲学与社会理论研究所);Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau (RPTU)(莱茵兰-普法尔茨州 Kaiserslautern-Landau 应用科技大学);German Research Centre for Artificial Intelligence (DFKI)(德国人工智能研究中心);Vilnius Gediminas Technical University(维尔纽斯格迪米纳斯技术大学);Drexel University(德雷塞尔大学);University of Montenegro(黑山大学);Faculty of Dramatic Arts(戏剧艺术学院);University of Kragujevac(克拉古耶瓦茨大学);Faculty of Engineering(工程学院)
Artificial Authority: From Machine Minds to Political Alignments. An Experimental Analysis of Democratic and Autocratic Biases in Large-Language Models
Jagiellonian University(雅盖隆大学);AGH University of Kraków(克拉科夫AGH大学);NASK National Research Institute(NASK国家研究院)