Large language models (LLMs) often exhibit strong biases, e.g, against women or in favor of the number 7. We investigate whether LLMs would be able to output less biased answers when allowed to observe their prior answers to the same question in a multi-turn conversation. To understand which types of questions invite more biased answers, we test LLMs on our proposed set of questions that span 9 topics and belong to three types: (1) Subjective; (2) Random; and (3) Objective. Interestingly, LLMs are able to "de-bias" themselves in a multi-turn conversation in response to questions that seek an Random, unbiased answer. Furthermore, we propose B-score, a novel metric that is effective in detecting biases to Subjective, Random, Easy, and Hard questions. On MMLU, HLE, and CSQA, leveraging B-score substantially improves the verification accuracy of LLM answers (i.e, accepting LLM correct answers and rejecting incorrect ones) compared to using verbalized confidence scores or the frequency of single-turn answers alone. Code and data are available at: https://b-score.github.io.
Are LLMs Rational Investors? A Study on Detecting and Reducing the Financial Bias in LLMs
School of Computer Science, Fudan University(复旦大学计算机科学技术学院);Institute of Fintech, Fudan University(复旦大学金融科技研究院);School of Electronics and Information Engineering, Tongji University(同济大学电子与信息工程学院);Tandon School of Engineering, New York University(纽约大学坦登工程学院);DataGrand Inc.(数据堂公司)
IndRegBias: A Dataset for Studying Indian Regional Biases in English and Code-Mixed Social Media Comments
Department of Data Science and Engineering, Indian Institute of Science Education and Research(数据科学与工程系,印度科学教育与研究学院);Oracle Industries AI, Oracle Corporation(Oracle人工智能部,Oracle公司)
Don't Erase, Inform! Detecting and Contextualizing Harmful Language in Cultural Heritage Collections
National Technical University of Athens(希腊国家技术大学);DFF - Deutsches Filminstitut & Filmmuseum(德国电影学院及电影博物馆);Europeana Foundation(欧洲ana基金会);Vrije Universiteit Amsterdam(阿姆斯特丹自由大学)
Old Habits Die Hard: How Conversational History Geometrically Traps LLMs
Technion - Israel Institute of Technology(技术ion-以色列理工学院);University of Oxford(牛津大学);University of Zagreb, FER(Zagreb大学,FER);Kempner Institute, Harvard University(Kempner研究所,哈佛大学);University of Edinburgh(爱丁堡大学)
A word association network methodology for evaluating implicit biases in LLMs compared to humans
University of Pisa, Department of Computer Science(比萨大学计算机科学系);National Research Council of Italy, Institute of Information Science and Technologies(意大利国家研究委员会信息科学与技术研究所);University of Trento, Department of Psychology and Cognitive Science(特伦托大学心理学与认知科学系)
From Bias to Balance: Detecting Facial Expression Recognition Biases in Large Multimodal Foundation Models
Lake Wales High School(莱克威尔士高中);Thomas Jefferson Sr. High School(托马斯·杰斐逊高中);Bellarmine College Preparatory(贝拉明学院预科学校);Algoverse AI Research(Algoverse AI研究机构)
A Systematic Review of Machine Learning Approaches for Detecting Deceptive Activities on Social Media: Methods, Challenges, and Biases
University of Pennsylvania(宾夕法尼亚大学);Northeastern University(东北大学);New York University(纽约大学);Georgia Institute of Technology(佐治亚理工学院);University of California, Berkeley(加州大学伯克利分校)
Detecting Political Biases of Named Entities and Hashtags on Twitter
University of California, Los Angeles(加利福尼亚大学洛杉矶分校);University of Southern California(南加利福尼亚大学);Information Sciences Institute(信息科学研究所);Santa Fe Institute(圣塔菲研究所)