Rating-based human evaluation has become an essential tool to accurately evaluate the impressive performance of large language models (LLMs). However, current rating systems suffer from several important limitations: first, they fail to account for biases that significantly influence evaluation results, second, they require large and expensive preference datasets to obtain accurate ratings, and third, they do not facilitate meaningful comparisons of model ratings across different tasks. To address these issues, we introduce Polyrating, an expressive and flexible rating system based on maximum a posteriori estimation that enables a more nuanced and thorough analysis of model performance at lower costs. Polyrating can detect and quantify biases affecting human preferences, ensuring fairer model comparisons. Further, Polyrating can reduce the cost of human evaluations by up to $41\%$ for new models and up to $77\%$ for new tasks by leveraging existing benchmark scores. Lastly, Polyrating enables direct comparisons of ratings across different tasks, providing a comprehensive understanding of an LLMs' strengths, weaknesses, and relative performance across different applications.
Improving Data Efficiency via Curating LLM-Driven Rating Systems
University of California, Santa Cruz(加州大学圣克鲁兹分校);Center for Advanced AI, Accenture(埃森哲高级人工智能中心);BIAI, ZJUT(浙江工业大学BIAI);D5Data.ai;The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect: Evidence from a Pre-Registered LLM-Judge Audit
Columbia University(哥伦比亚大学);Johns Hopkins University(约翰斯·霍普金斯大学);The University of Chicago(芝加哥大学);The Pennsylvania State University(宾夕法尼亚州立大学);The Hong Kong Polytechnic University(香港理工大学)
The Biased Oracle: Assessing LLMs' Understandability and Empathy in Medical Diagnoses
Department of Chemistry and Applied Biosciences, ETH Zurich(应用生物科学系,苏黎世联邦理工学院);Department of Computer Science, ETH Zurich(计算机科学系,苏黎世联邦理工学院);Department of Health Sciences and Technology, ETH Zurich(健康科学与技术系,苏黎世联邦理工学院)
Machine individuality: Separating genuine idiosyncrasy from response bias in large language models
Center for Adaptive Rationality, Max Planck Institute for Human Development(适应性理性中心,马克斯·普朗克人类发展研究所);Center for Cognitive and Decision Sciences, Department of Psychology, University of Basel(认知与决策科学中心,心理学系,巴塞尔大学)
Large Language Models are biased to overestimate profoundness
Pontificia Universidad Católica de Chile(智利天主教大学);Centro Nacional de Inteligencia Artificial (CENIA)(国家人工智能中心);Brown University(布朗大学);Leiden University(莱顿大学)
Automated scoring of the Ambiguous Intentions Hostility Questionnaire using fine-tuned large language models
University of California, Los Angeles(加州大学洛杉矶分校);University of Texas at Tyler(德克萨斯大学泰勒分校);JFK Johnson Rehabilitation Institute(肯尼迪约翰逊康复研究所);University of Chicago(芝加哥大学)
The Voice Behind the Words: Quantifying Intersectional Bias in SpeechLLMs
1 Department of Speech, Music;Hearing, KTH Royal Institute of Technology, Sweden 2 Centre for Speech Technology Research, University of Edinburgh, UK 3 Texas A\&M University, USA
The Effect of Human v/s Synthetic Test Data and Round-tripping on Assessment of Sentiment Analysis Systems for Bias
AI Institute University of South Carolina(南卡罗来纳大学人工智能研究所);Dept. of Computer Science Netaji Subhas University of Technology(Netaji Subhas 科技大学计算机科学系);Dept. of Computer Science and Engineering University of South Carolina(南卡罗来纳大学计算机科学与工程系);Dept. of Integrated Information Technology University of South Carolina(南卡罗来纳大学综合信息技术系)