Large Language Models (LLMs) inherently carry the biases contained in their training corpora, which can lead to the perpetuation of societal harm. As the impact of these foundation models grows, understanding and evaluating their biases becomes crucial to achieving fairness and accuracy. We propose to study what LLMs know about the world we live in through the lens of geography. This approach is particularly powerful as there is ground truth for the numerous aspects of human life that are meaningfully projected onto geographic space such as culture, race, language, politics, and religion. We show various problematic geographic biases, which we define as systemic errors in geospatial predictions. Initially, we demonstrate that LLMs are capable of making accurate zero-shot geospatial predictions in the form of ratings that show strong monotonic correlation with ground truth (Spearman's $ρ$ of up to 0.89). We then show that LLMs exhibit common biases across a range of objective and subjective topics. In particular, LLMs are clearly biased against locations with lower socioeconomic conditions (e.g. most of Africa) on a variety of sensitive subjective topics such as attractiveness, morality, and intelligence (Spearman's $ρ$ of up to 0.70). Finally, we introduce a bias score to quantify this and find that there is significant variation in the magnitude of bias across existing LLMs. Code is available on the project website: https://rohinmanvi.github.io/GeoLLM
Documenting Geographically and Contextually Diverse Data Sources: The BigScience Catalogue of Language Data and Resources
Booz Allen Hamilton（博思艾伦咨询公司）；British Library（大英图书馆）；EleutherAI；Hugging Face；Inria（法国国家信息与自动化研究所）；University of Washington（华盛顿大学）；University of Western Australia（西澳大利亚大学）；University of the Basque Country（巴斯克大学）；KFUPM（法赫德国王石油矿产大学）
Optimizing Resource Allocation for Geographically-Distributed Inference by Large Language Models
Department of Computer Science and Engineering, Pennsylvania State University, University Park, PA, USA（计算机科学与工程系，宾夕法尼亚州立大学，大学公园，PA，USA）；Department of Computer Science, Virginia Tech, Blacksburg, VA, USA（计算机科学系，弗吉尼亚理工大学，布莱克堡，VA，USA）；Department of Electrical Communication Engineering, Indian Institute of Science, Bangalore, India（电子通信工程系，印度科学研究院，班加罗尔，印度）
Using Zero-Shot LLM-Generated Survey Data for Geographically Explicit Population Synthesis
Dept. of Geography and Geoinformation Science, George Mason University, Fairfax, VA, USA（地理与地理信息科学系，乔治·马歇尔大学，弗吉尼亚州 Fairfax）；Dept. of Computer Science, George Mason University, Fairfax, VA, USA（计算机科学系，乔治·马歇尔大学，弗吉尼亚州 Fairfax）；College of Public Health, George Mason University, Fairfax, VA, USA（公共卫生学院，乔治·马歇尔大学，弗吉尼亚州 Fairfax）；Dept. of Geography, University at Buffalo, Buffalo, NY, USA（地理系，布法罗大学，纽约州 Buffalo）；Dept. of Computational and Data Sciences, George Mason University, Fairfax, VA, USA（计算与数据科学系，乔治·马歇尔大学，弗吉尼亚州 Fairfax）
Emory University（埃默里大学）；Indian Institute of Technology Roorkee（印度理工学院罗里克分校）；Cairo University（开罗大学）；Mila - Quebec AI Institute（魁北克人工智能研究所）；Polytechnique Montréal（蒙特利尔理工学院）；Amazon（亚马逊）
Unraveling Media Perspectives: A Comprehensive Methodology Combining Large Language Models, Topic Modeling, Sentiment Analysis, and Ontology Learning to Analyse Media Bias
Institute of IT Management and Digitization Research (IFID)（信息管理与数字化研究学院）
Where Should I Study? Biased Language Models Decide! Evaluating Fairness in LMs for Academic Recommendations
Centre for Responsible AI (CeRAI), Wadhwani School of Data Science and AI (WSAI)（负责任人工智能中心（CeRAI）、Wadhwani数据科学与人工智能学校（WSAI））；Indian Institute of Technology Madras（印度理工学院马德拉斯学院）
Fake News Detectors are Biased against Texts Generated by Large Language Models
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Monash University（莫纳什大学）；CSIRO’s Data61（澳大利亚联邦科学与工业研究组织Data61部门）
Large Language Models are biased to overestimate profoundness
Pontificia Universidad Católica de Chile（智利天主教大学）；Centro Nacional de Inteligencia Artificial (CENIA)（国家人工智能中心）；Brown University（布朗大学）；Leiden University（莱顿大学）
BIASEDTALES-ML: A Multilingual Dataset for Analyzing Narrative Attribute Distributions in LLM-Generated Stories
School of Computer Science and Engineering, Northeastern University, Shenyang, China（东北大学计算机科学与工程学院，中国沈阳）；NiuTrans Research, Shenyang, China（NiuTrans研究院，中国沈阳）
Fairness Testing of Large Language Models in Role-Playing
Peking University（北京大学）；Tsinghua University（清华大学）；King's College London（伦敦国王学院）；Nanyang Technological University（南洋理工大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought
Speechmatics, Apollo Research（Speechmatics与Apollo研究机构）；University of Oxford（牛津大学）；NYU, Anthropic（纽约大学与Anthropic）；NYU（纽约大学）；Anthropic, NYU（Anthropic与纽约大学）
CARTE: A Benchmark for Mapping Language Model Knowledge Across France
École Polytechnique, Institut Polytechnique de Paris（巴黎政治学院）；National Technical University of Athens（雅典国家技术大学）；Mohamed bin Zayed University of Artificial Intelligence（姆阿扎德人工智能大学）
The Biased Samaritan: LLM biases in Perceived Kindness
Department of Cognitive Science University of California, Davis（认知科学系加州大学戴维斯分校）；Department of Statistics University of California, Davis（统计学系加州大学戴维斯分校）；Department of Computer Science University of California, Davis（计算机科学系加州大学戴维斯分校）
Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Peking University（北京大学）；Tencent Inc（腾讯公司）
