发表机构
Indian Institute of Science（印度科学学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
BiG-SURE是一种基于跨温度语义一致性的二分图方法，用于黑盒LLMs/VLMs的语义不确定性与可靠性估计，在多类问答任务上提升了弃权AUROC性能。
AI 中文摘要
可靠的不确定性估计是将大语言模型（LLMs）和视觉语言模型（VLMs）部署到安全关键场景的关键要求，尤其是在无法访问模型参数（黑盒设置）时。我们提出BiG-SURE，一种基于跨温度语义一致性的不确定性估计方法。该方法在保持语义的输入变换下，采样低温响应作为稳定语义锚点，高温响应作为探针，随后使用基于自然语言推理（NLI）的蕴含分数构建锚点-探针二分图（BiG），并通过该矩阵的归一化平方谱能量定义置信度，不确定性则为其补集。这种基于二分图的语义不确定性与可靠性估计（SURE）分数用于衡量高温探针是否与模型稳定的低温信念保持语义对齐。我们在多个模型家族的文本问答、多语言问答和多模态问答任务上评估BiG-SURE，实验结果显示，BiG-SURE在弃权（不执行）AUROC指标上优于现有黑盒不确定性估计器，同时保持简单、无监督且适用于黑盒模型设置的特性。
英文摘要
Reliable uncertainty estimation is a crucial requirement for deploying large language models (LLMs) and vision-language models (VLMs) in safety-critical settings, especially when the model parameters are not accessible (black-box). We propose BiG-SURE, an uncertainty estimator based on cross-temperature semantic agreement. The method samples low-temperature responses as stable semantic anchors and high-temperature responses as probes under meaning-preserving input transformations. It then constructs an anchor-probe Bipartite Graph (BiG) using NLI-based entailment scores and defines confidence through the normalized squared spectral energy of this matrix, with uncertainty given by its complement. This bipartite graph-based Semantic Uncertainty and Reliability Estimation (SURE) score measures whether high-temperature probes remain semantically aligned with the model's stable low-temperature belief or not. We evaluate BiG-SURE on text QA, multilingual QA, and multimodal QA tasks across multiple model families. In these experiments, BiG-SURE improves average abstention AUROC over prior black-box uncertainty estimators, while remaining simple, unsupervised, and applicable to black-box model settings.
Comments22 pages, 9 figures
Journal refEMNLP 2026 Main Conference
