UBench: Benchmarking Uncertainty in Large Language Models with Multiple Choice Questions
College of Software, Nankai University(南开大学软件学院);Institute for Advanced Algorithms Research (Shanghai)(上海先进算法研究所);College of Artificial Intelligence, Tianjin University of Science and Technology(天津科技大学人工智能学院);Tianjin Key Laboratory of Software Experience and Human Computer Interaction(天津软件体验与人机交互重点实验室)
Xunzhi Wang, Zhuowei Zhang, Gaonan Chen, Qiongyu Li, Bitong Luo, Zhixin Han, Haotian Wang, Zhiyu li, Hang Gao, Mengting Hu
Despite recent progress in systematic evaluation frameworks, benchmarking the uncertainty of large language models (LLMs) remains a highly challenging task. Existing methods for benchmarking the uncertainty of LLMs face three key challenges: the need for internal model access, additional training, or high computational costs. This is particularly unfavorable for closed-source models. To this end, we introduce UBench, a new benchmark for evaluating the uncertainty of LLMs. Unlike other benchmarks, UBench is based on confidence intervals. It encompasses 11,978 multiple-choice questions spanning knowledge, language, understanding, and reasoning capabilities. Based on this, we conduct extensive experiments. This includes comparisons with other advanced uncertainty estimation methods, the assessment of the uncertainty of 20 LLMs, and an exploration of the effects of Chain-of-Thought (CoT) prompts, role-playing (RP) prompts, and temperature on model uncertainty. Our analysis reveals several crucial insights: 1) Our confidence interval-based methods are highly effective for uncertainty quantification; 2) Regarding uncertainty, outstanding open-source models show competitive performance versus closed-source models; 3) CoT and RP prompts present potential ways to improve model reliability, while the influence of temperature changes follows no universal rule. Our implementation is available at https://github.com/Cyno2232/UBENCH.
D2VBench: Benchmarking Large Language Models with Value Dilemmas in Daily Scenarios
TJUNLP Lab, School of Computer Science and Technology, Tianjin University(天津大学计算机科学与技术学院TJUNLP实验室);The International Joint Institute of Tianjin University(天津大学国际联合研究院)
Vote'n'Rank: Revision of Benchmarking with Social Choice Theory
HSE University(高等经济大学);SaluteDevices(SaluteDevices公司);University of Oxford(牛津大学);Artificial Intelligence Research Institute(人工智能研究院);Institute of Linguistics, RAS(俄罗斯科学院语言学研究所);Center for Information and Language Processing, LMU Munich(慕尼黑大学信息与语言处理中心)
LEXam: Benchmarking Legal Reasoning on 340 Law Exams
ETH Zurich(苏黎世联邦理工学院);University of Zurich(苏黎世大学);University of Lausanne(洛桑大学);Max Planck Institute for Research on Collective Goods(马克斯·普朗克集体物品研究所);Omnilex;University of St. Gallen(圣加仑大学);Swiss Federal Supreme Court(瑞士联邦最高法院)
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code
Allen Institute for AI(艾伦人工智能研究所);Amazon Alexa AI(亚马逊Alexa人工智能部门);Bangladesh University of Engineering and Technology(孟加拉国工程技术大学);Carnegie Mellon University(卡内基梅隆大学);Charles University(查理大学)