arXiv 2603.180092026-03cs.CLcs.AI

第一个token的可信度如何？一种用于大型语言模型分类和理解的不确定性校准提示优化框架

How Confident Is the First Token? An Uncertainty-Calibrated Prompt Optimization Framework for Large Language Model Classification and Understanding

China Jiliang University（中国嘉兴大学）

Wei Chen, Guoyang Ju, Yuanyuan Qi

备注：: 27 pages,16 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出LSFU指标和UCPOF框架，通过校准提示优化提升模型性能，实验显示在少样本基准上准确率提升6.03%，超越全RAG方法，并降低检索触发率50.66%。

摘要：

随着大型语言模型（LLMs）在自然语言处理中的广泛应用，提示工程和检索增强生成（RAG）已成为提升LLMs在复杂任务上性能的主流方法。然而，LLMs生成输出是自回归的，导致不可避免的输出不确定性。由于模型性能对提示设计高度敏感，精确的不确定性测量对可靠的提示优化至关重要。对于多类多项选择（理解）任务，传统不确定性度量（如熵）基于输出概率，将所有类别视为平等，忽略了预训练语料库中类先验差异。这种无法区分虚假自信（来自先验）与真实确定性（来自上下文理解）导致信心校准不佳。为此，我们提出Log-Scale Focal Uncertainty（LSFU），一种基于第一个token的指标，受焦点损失启发。LSFU将标签先验概率作为风险调节因子，以抑制高频类别的噪声并强调低频长尾类别的风险，通过动态加权机制统一测量尺度。基于LSFU，我们进一步提出不确定性校准的提示优化框架（UCPOF），利用模型输出的第一个token选择高质量示例并动态优化提示。全面评估显示，UCPOF在少样本基准上将平均准确率提高6.03%，在整体平均准确率上超越始终开启的全RAG方法5.75%，并减少平均检索触发率50.66%。通过适应性触发RAG仅针对高不确定性样本，我们的框架显著降低了计算成本，同时保持了最先进的性能。

英文摘要：

With the widespread adoption of large language models (LLMs) in natural language processing, prompt engineering and retrieval-augmented generation (RAG) have become mainstream to enhance LLMs' performance on complex tasks. However, LLMs generate outputs autoregressively, leading to inevitable output uncertainty. Since model performance is highly sensitive to prompt design, precise uncertainty measurement is crucial for reliable prompt optimization. For multi-class multiple-choice (understanding) tasks, conventional uncertainty measures (e.g., entropy) based on output probabilities treat all classes equally and ignore class prior differences in pretraining corpora. This failure to distinguish spurious confidence (from priors) from true certainty (from contextual understanding) results in poor confidence calibration. To address this, we propose Log-Scale Focal Uncertainty (LSFU), a first-token-based metric inspired by focal loss. LSFU incorporates label prior probabilities as a risk-modulation factor to suppress noise from high-frequency classes and emphasize risk for low-frequency long-tail classes, with a dynamic weighting mechanism unifying the measurement scale. Based on LSFU, we further propose the uncertainty-calibrated prompt optimization framework (UCPOF), which leverages the first token of model outputs to select high-quality exemplars and dynamically optimize prompts. Comprehensive evaluations show UCPOF improves average accuracy by 6.03% over few-shot baselines, surpasses always-on full RAG by 5.75% in overall average accuracy, and reduces the average retrieval trigger rate by 50.66%. By adaptively triggering RAG only for high-uncertainty samples, our framework significantly lowers computational costs while maintaining state-of-the-art performance.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文