MCQA-Eval：基于金标准正确性标签的自然语言生成高效置信度评估
MCQA-Eval: Efficient Confidence Evaluation in NLG with Gold-Standard Correctness Labels
AI总结：
针对现有NLG置信度评估框架依赖有噪声、成本高的启发式正确性函数的问题，提出MCQA-Eval框架，利用多项选择数据集金标准标签实现白盒与黑盒置信度度量的统一高效可靠评估。
摘要：
大语言模型（LLM）需要鲁棒的置信度估计，尤其是在医疗、法律等关键领域，不可靠的输出可能引发严重后果。尽管近期已有大量置信度估计相关研究，但当前评估框架依赖正确性函数——这类启发式方法通常存在噪声、成本高昂，还可能引入系统性偏差。这些方法学缺陷往往会扭曲评估指标，进而影响置信度度量方法的对比排名。我们提出MCQA-Eval，一种用于评估自然语言生成（NLG）中置信度度量的评估框架，它利用来自多项选择数据集的金标准正确性标签，消除了对显式正确性函数的依赖。MCQA-Eval可对基于内部状态的白盒（如基于logit的）和基于一致性的黑盒置信度度量开展系统性对比，为不同类型的方法提供统一的评估范式。通过在多个大语言模型和常用QA数据集上开展大量实验，我们发现MCQA-Eval相比现有方法，能对置信度估计方法提供更高效、更可靠的评估。
英文摘要：
Large Language Models (LLMs) require robust confidence estimation, particularly in critical domains like healthcare and law where unreliable outputs can lead to significant consequences. Despite much recent work in confidence estimation, current evaluation frameworks rely on correctness functions -- various heuristics that are often noisy, expensive, and possibly introduce systematic biases. These methodological weaknesses tend to distort evaluation metrics and thus the comparative ranking of confidence measures. We introduce MCQA-Eval, an evaluation framework for assessing confidence measures in Natural Language Generation (NLG) that eliminates dependence on an explicit correctness function by leveraging gold-standard correctness labels from multiple-choice datasets. MCQA-Eval enables systematic comparison of both internal state-based white-box (e.g. logit-based) and consistency-based black-box confidence measures, providing a unified evaluation methodology across different approaches. Through extensive experiments on multiple LLMs and widely used QA datasets, we report that MCQA-Eval provides efficient and more reliable assessments of confidence estimation methods than existing approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
