arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.19684cs.CL
收藏

GRACE:一个用于评估模型校准与人类校准对比的细粒度基准

GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibration

University of Maryland(马里兰大学);UC Berkeley(加州大学伯克利分校);IIT Bombay(印度理工学院孟买分校)

Yoo Yeon Sung, Eve Fleisig, Yu Hou, Ishan Upadhyay, Jordan Lee Boyd-Graber

AI总结:

GRACE是一个细粒度基准,通过线索问答对比人类与模型的校准,提出CalScore指标,发现人类虽准确性低但校准更好,用于评估模型校准改进。

摘要:

语言模型常常校准不佳,导致出现自信但错误的答案。我们引入了GRACE,一个用于语言模型校准的基准,该基准纳入了与人类校准的对比。GRACE由问答对组成,每个问题包含一系列逐渐变容易的线索,这些线索都指向同一个答案;随着线索的逐步揭示,模型必须尽可能早地正确作答。这一设置允许基于模型作答的早晚、准确性和置信度来细粒度地测量模型校准。在收集这些问题后,我们举办了人类与模型的实时竞赛,收集了关于人类和模型团队在时间、准确性和置信度方面的1,749个数据点。我们提出了一种度量指标CalScore,利用GRACE来分析模型校准误差,并识别出与人类行为不同的模型校准错误类型。我们发现,尽管人类在准确性上不如模型,但人类通常校准得更好。由于最先进的模型在GRACE上表现困难,该基准能有效评估改进模型校准方面的进展。

英文摘要:

Language models are often miscalibrated, leading to confidently incorrect answers. We introduce GRACE, a benchmark for language model calibration that incorporates comparison with human calibration. GRACE consists of question-answer pairs, in which each question contains a series of clues that gradually become easier, all leading to the same answer; models must answer correctly as early as possible as the clues are revealed. This setting permits granular measurement of model calibration based on how early, accurately, and confidently a model answers. After collecting these questions, we host live human vs. model competitions to gather 1,749 data points on human and model teams' timing, accuracy, and confidence. We propose a metric, CalScore, that uses GRACE to analyze model calibration errors and identify types of model miscalibration that differ from human behavior. We find that although humans are less accurate than models, humans are generally better calibrated. Since state-of-the-art models struggle on GRACE, it effectively evaluates progress on improving model calibration.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑