发表机构
Layer 6 AI; TD Insurance（Layer 6 AI; TD保险）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究针对黑盒大型语言模型的不确定性量化问题，利用目标数据集构建基于现有分数和相似查询正确性的简单分类器，以改进置信度估计，且计算开销极小。
AI 中文摘要
不确定性量化（UQ）对于大型语言模型（LLM）的安全部署至关重要。现有方法包括口头置信度评估及需要多次生成的方法，通常为零样本方法，无需标注数据即可生成量化不确定性的分数。但实践中，部署前必须在目标数据集上评估其性能。本研究表明，利用该目标数据集可使我们的方法持续优于现有分数。具体而言，我们构建简单分类器，将现有分数及相似查询的正确性作为特征，预测LLM响应的正确性。我们的方法计算开销极小，是LLM不确定性量化（UQ）用于实际应用的廉价且直接的增强方案。
英文摘要
Uncertainty quantification (UQ) is essential for the safe deployment of large language models (LLMs). Existing methods, from verbalized confidence to ones requiring multiple generations, are often zero-shot and produce scores quantifying uncertainty without the need for labelled data. Nonetheless, in practice one must always evaluate their performance on a dataset of interest before deployment. In this work we show that, by leveraging this dataset, we consistently outperform these existing scores. Specifically, we build simple classifiers that predict LLM response correctness by using these scores and the correctness of similar queries as features. Our method produces minimal computational overhead, making it a cheap and straightforward enhancement for UQ in LLMs for real-world applications.
