arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.03344cs.CL
收藏

薛定谔的阈值:当AUC无法预测准确率时

Schroedinger's Threshold: When the AUC doesn't predict Accuracy

University of Zurich(苏黎世大学)

Juri Opitz

AI总结:

本文揭示AUC在评估文本忠实度预测模型时可能高估准确率并改变基准排名,通过探索不同校准模式以更真实反映下游性能。

摘要:

曲线下面积(AUC)度量似乎适合评估和比较各种模型,且可能无需校准。AUC应用的一个重要示例是评估和基准测试预测生成文本忠实度的模型。但我们表明,AUC产生了一种学术且乐观的准确率概念,可能与应用中观察到的实际准确率不一致,导致基准排名发生显著变化。为了更真实地描绘下游模型性能(并为实际应用准备模型),我们探索了不同的校准模式,测试了校准数据和方法。

英文摘要:

The Area Under Curve measure (AUC) seems apt to evaluate and compare diverse models, possibly without calibration. An important example of AUC application is the evaluation and benchmarking of models that predict faithfulness of generated text. But we show that the AUC yields an academic and optimistic notion of accuracy that can misalign with the actual accuracy observed in application, yielding significant changes in benchmark rankings. To paint a more realistic picture of downstream model performance (and prepare a model for actual application), we explore different calibration modes, testing calibration data and method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑