语义自蒸馏用于语言模型不确定性
Semantic Self-Distillation for Language Model Uncertainty
AI总结：
本文提出语义自蒸馏方法，通过蒸馏生成轻量模型以估计提示条件密度，提供提示级不确定性信号和答案级可靠性评估，在TriviaQA和MMLU实验中表现出竞争力。
摘要：
大型语言模型在原理上难以量化不确定性，部分原因在于其复杂性和输出多样性。语义分散，即采样答案意义的方差，已被提出作为模型不确定性的有用代理，但相关的计算成本限制了其在延迟敏感应用中的使用。我们显示，采样语义分布可以蒸馏成轻量学生模型，在语言模型生成答案标记之前估计提示条件密度。学生模型预测可能答案的语义分布；该分布的熵提供提示级不确定性信号，概率密度允许答案级可靠性评估。在TriviaQA和MMLU实验中，我们发现学生模型在幻觉预测任务中相对于基于采样的语义分散基线表现竞争，同时提供额外的不确定性原语用于领域外检测和多选答案选择。我们将此技术称为语义自蒸馏（SSD），它可以作为在复杂输出空间中蒸馏预测不确定性的通用框架。
英文摘要：
Large language models present challenges for principled uncertainty quantification, in part due to their complexity and the diversity of their outputs. Semantic dispersion, or the variance in the meaning of sampled answers, has been proposed as a useful proxy for model uncertainty, but the associated computational cost prohibits its use in latency-critical applications. We show that sampled semantic distributions can be distilled into lightweight student models which estimate a prompt-conditioned density before the language model generates an answer token. The student model predicts a semantic distribution over possible answers; the entropy of this distribution provides a prompt-level uncertainty signal, and the probability density allows answer-level reliability evaluation. Across experiments on TriviaQA and MMLU, we find our student models perform competitively relative to the teacher's sampled semantic dispersion on a hallucination prediction task, whilst offering additional uncertainty primitives for out-of-domain detection and multiple-choice answer selection. We term this technique Semantic Self-Distillation (SSD), which can serve as a general framework for distilling predictive uncertainty in complex output spaces beyond language.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
