arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.01122cs.CLcs.LG
收藏

用于二分类问答的校准大语言模型

Calibrated Large Language Models for Binary Question Answering

Royal Holloway, University of London(伦敦大学皇家霍洛威学院);Centrica(森特里卡公司)

Patrizio Giovannotti, Alexander Gammerman

AI总结:

本文提出利用归纳Venn–Abers预测器(IVAP)校准大语言模型在二分类问答中的输出概率,并在BoolQ数据集上验证其优于温度缩放方法,同时保持高预测质量。

摘要:

量化大语言模型(LLMs)在二分类文本分类任务中预测的不确定性仍然是一个挑战。在LLMs的语境下,校准指的是模型预测概率与其预测实际正确性之间的一致性。一个校准良好的模型应产生能够准确反映其预测正确可能性的概率。我们提出了一种新方法,利用归纳Venn–Abers预测器(IVAP)来校准与二分类标签对应的输出token相关的概率。我们在BoolQ数据集上使用Llama 2模型进行的实验表明,对于各种标签token选择,IVAP始终优于常用的温度缩放方法,在保持高预测质量的同时实现了校准良好的概率。我们的发现有助于理解LLMs的校准技术,并为在二分类问答任务中获得可靠的置信度估计提供了实用解决方案,从而增强了LLM预测的可解释性和可信度。

英文摘要:

Quantifying the uncertainty of predictions made by large language models (LLMs) in binary text classification tasks remains a challenge. Calibration, in the context of LLMs, refers to the alignment between the model's predicted probabilities and the actual correctness of its predictions. A well-calibrated model should produce probabilities that accurately reflect the likelihood of its predictions being correct. We propose a novel approach that utilizes the inductive Venn--Abers predictor (IVAP) to calibrate the probabilities associated with the output tokens corresponding to the binary labels. Our experiments on the BoolQ dataset using the Llama 2 model demonstrate that IVAP consistently outperforms the commonly used temperature scaling method for various label token choices, achieving well-calibrated probabilities while maintaining high predictive quality. Our findings contribute to the understanding of calibration techniques for LLMs and provide a practical solution for obtaining reliable uncertainty estimates in binary question answering tasks, enhancing the interpretability and trustworthiness of LLM predictions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑