arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.12784cs.CL
收藏

UBench:利用多项选择题对大语言模型的不确定性进行基准测试

UBench: Benchmarking Uncertainty in Large Language Models with Multiple Choice Questions

College of Software, Nankai University(南开大学软件学院);Institute for Advanced Algorithms Research (Shanghai)(上海先进算法研究所);College of Artificial Intelligence, Tianjin University of Science and Technology(天津科技大学人工智能学院);Tianjin Key Laboratory of Software Experience and Human Computer Interaction(天津软件体验与人机交互重点实验室)

Xunzhi Wang, Zhuowei Zhang, Gaonan Chen, Qiongyu Li, Bitong Luo, Zhixin Han, Haotian Wang, Zhiyu li, Hang Gao, Mengting Hu

AI总结:

UBench提出基于置信区间的LLM不确定性基准,包含11,978道多选题,通过大规模实验验证其方法有效性,并揭示开源模型竞争力及提示策略对不确定性的影响。

摘要:

尽管近年来系统性评估框架取得了进展,但对大语言模型(LLMs)的不确定性进行基准测试仍然是一项极具挑战性的任务。现有的LLM不确定性基准测试方法面临三个关键挑战:需要访问模型内部、需要额外训练,或计算成本高昂。这对闭源模型尤其不利。为此,我们提出了UBench,一个用于评估LLM不确定性的新基准。与其他基准不同,UBench基于置信区间构建。它包含11,978道多项选择题,覆盖知识、语言、理解和推理能力。在此基础上,我们进行了大量实验,包括与其他先进不确定性估计方法的比较、对20个LLM不确定性的评估,以及探索思维链(CoT)提示、角色扮演(RP)提示和温度对模型不确定性的影响。我们的分析揭示了几个关键发现:1)我们基于置信区间的方法在不确定性量化方面非常有效;2)在不确定性方面,优秀的开源模型表现出与闭源模型相当的竞争力;3)CoT和RP提示为提升模型可靠性提供了潜在途径,而温度变化的影响并不遵循普遍规律。我们的实现可在 https://github.com/Cyno2232/UBENCH 获取。

英文摘要:

Despite recent progress in systematic evaluation frameworks, benchmarking the uncertainty of large language models (LLMs) remains a highly challenging task. Existing methods for benchmarking the uncertainty of LLMs face three key challenges: the need for internal model access, additional training, or high computational costs. This is particularly unfavorable for closed-source models. To this end, we introduce UBench, a new benchmark for evaluating the uncertainty of LLMs. Unlike other benchmarks, UBench is based on confidence intervals. It encompasses 11,978 multiple-choice questions spanning knowledge, language, understanding, and reasoning capabilities. Based on this, we conduct extensive experiments. This includes comparisons with other advanced uncertainty estimation methods, the assessment of the uncertainty of 20 LLMs, and an exploration of the effects of Chain-of-Thought (CoT) prompts, role-playing (RP) prompts, and temperature on model uncertainty. Our analysis reveals several crucial insights: 1) Our confidence interval-based methods are highly effective for uncertainty quantification; 2) Regarding uncertainty, outstanding open-source models show competitive performance versus closed-source models; 3) CoT and RP prompts present potential ways to improve model reliability, while the influence of temperature changes follows no universal rule. Our implementation is available at https://github.com/Cyno2232/UBENCH.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑