只需要求校准：从基于人类反馈微调的语言模型中引出校准置信度分数的策略
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback
AI总结：
本文评估了从RLHF微调的语言模型中提取置信度的方法，发现口头化置信度比模型条件概率校准更好，在多个基准上将预期校准误差相对降低50%。
摘要：
一个值得信赖的真实世界预测系统应该产生校准良好的置信度分数；也就是说，其对答案的置信度应该指示答案正确的可能性，从而在低置信度预测的情况下允许移交给专家。最近的研究表明，无监督预训练产生的大型语言模型（LMs）的条件概率校准得非常好。然而，最广泛使用的LMs是基于人类反馈的强化学习（RLHF-LMs）进行微调的，一些研究表明RLHF-LMs产生的条件概率校准非常差。鉴于这一已察觉的弱点，我们对从RLHF-LMs中提取置信度分数的方法进行了广泛评估。对于ChatGPT、GPT-4和Claude等RLHF-LMs，我们发现作为输出token发出的口头化置信度通常比模型在TriviaQA、SciQ和TruthfulQA基准上的条件概率校准得更好，通常将预期校准误差相对降低50%。
英文摘要：
A trustworthy real-world prediction system should produce well-calibrated confidence scores; that is, its confidence in an answer should be indicative of the likelihood that the answer is correct, enabling deferral to an expert in cases of low-confidence predictions. Recent studies have shown that unsupervised pre-training produces large language models (LMs) whose conditional probabilities are remarkably well-calibrated. However, the most widely-used LMs are fine-tuned with reinforcement learning from human feedback (RLHF-LMs), and some studies have suggested that RLHF-LMs produce conditional probabilities that are very poorly calibrated. In light of this perceived weakness, we conduct a broad evaluation of methods for extracting confidence scores from RLHF-LMs. For RLHF-LMs such as ChatGPT, GPT-4, and Claude, we find that verbalized confidences emitted as output tokens are typically better-calibrated than the model's conditional probabilities on the TriviaQA, SciQ, and TruthfulQA benchmarks, often reducing the expected calibration error by a relative 50%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
