Recent advances in large language models (LLMs) show the potential of using LLMs as evaluators for assessing the quality of text generations from LLMs. However, applying LLM evaluators naively to compare or judge between different systems can lead to unreliable results due to the intrinsic win rate estimation bias of LLM evaluators. In order to mitigate this problem, we propose two calibration methods, Bayesian Win Rate Sampling (BWRS) and Bayesian Dawid-Skene, both of which leverage Bayesian inference to more accurately infer the true win rate of generative language models. We empirically validate our methods on six datasets covering story generation, summarization, and instruction following tasks. We show that both our methods are effective in improving the accuracy of win rate estimation using LLMs as evaluators, offering a promising direction for reliable automatic text quality evaluation.
Personalized Turn-Level User Conversation Satisfaction Benchmark
Department of Computer Science and Technology, Tsinghua University, Beijing, China.（清华大学计算机科学与技术系，北京，中国）；Institute for AI Industry Research, Tsinghua University, Beijing, China.（清华大学人工智能产业研究院，北京，中国）；Meituan（美团）
The Fellowship of the LLMs: Multi-Model Workflows for Synthetic Preference Optimization Dataset Generation
Lahore University of Management Sciences（拉合尔管理科学大学）；University of Michigan - Ann Arbor（密歇根大学安娜堡分校）；EMBL European Bioinformatics Institute（欧洲生物信息学研究所）；Strategize Inc（Strategize公司）
DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation
National University of Singapore（新加坡国立大学）；Salesforce Research（Salesforce研究院）；Nanyang Technological University（南洋理工大学）；Institute for Infocomm Research (IR)（信息通信研究院）；A*STAR（新加坡科技研究局）
University of Arizona, USA（亚利桑那大学）；Arizona State University, USA（亚利桑那州立大学）；Now at Google LLC, work done at Rice University（现就职于谷歌公司，曾就职于里士大学）；Clemson University, USA（克莱姆森大学）；Washington University in St. Louis, USA（圣路易斯华盛顿大学）；Halmstad University, Sweden（哈姆斯塔德大学）；Guangdong Institute of Intelligence Science and Technology, China（广东智能科学与技术研究院）
