具有可证明风险控制的多项选择题回答任务中的保形 P 值
Conformal P-Value in Multiple-Choice Question Answering Tasks with Provable Risk Control
AI总结：
本研究提出一种结合显著性检验与保形预测的框架，通过自洽性重采样和 p 值计算构建预测集，有效缓解 LLMs 在多项选择题回答中的幻觉问题，并验证了其风险控制与不确定性度量能力。
摘要：
本研究引入了一种显著性检验增强的保形预测（CP）框架，以提高大型语言模型（LLMs）在多项选择题回答（MCQA）中的可信度。尽管 LLMs 越来越多地被部署于学科 QA 场景，但幻觉和非事实生成极大地损害了响应的可靠性。虽然 CP 为预测集提供了统计上严格的边际覆盖保证，且显著性检验提供了既定的统计严谨性，但它们的协同整合仍有待探索。为了缓解幻觉和事实不准确，我们的框架通过对 MCQA 响应进行自洽性重采样，将 $p$ 值计算与一致性评分相结合。该方法计算选项频率以应对 LLMs 的黑盒性质，随后利用经验推导的 $p$ 值通过零假设检验（$\mathcal{H}_0$）构建预测集。在 MMLU 和 MMLU-Pro 基准上使用现成 LLMs 进行的评估表明：（1）增强的 CP 实现了用户指定的经验未覆盖率；（2）测试集平均预测集大小（APSS）随风险水平（$α$）的增加而单调递减，验证了 APSS 作为有效不确定性指标的适用性。这项工作为在高风险 QA 应用中部署可信 LLM 建立了有原则的统计框架。
英文摘要：
This study introduces a significance testing-enhanced conformal prediction (CP) framework to improve trustworthiness of large language models (LLMs) in multiple-choice question answering (MCQA). While LLMs have been increasingly deployed in disciplinary QA scenarios, hallucination and nonfactual generation substantially compromise response reliability. Although CP provides statistically rigorous marginal coverage guarantees for prediction sets, and significance testing offers established statistical rigor, their synergistic integration remains unexplored. To mitigate hallucination and factual inaccuracies, our framework integrates $p$-value computation with conformity scoring through self-consistency resampling of MCQA responses. This approach calculates option frequencies to address LLMs' black-box nature, subsequently constructing prediction sets via null hypothesis testing ($\mathcal{H}_0$) with empirically derived $p$-values. Evaluations on MMLU and MMLU-Pro benchmarks using off-the-shelf LLMs demonstrate: (1) The enhanced CP achieves user-specified empirical miscoverage rates; (2) Test-set average prediction set size (APSS) decreases monotonically with increasing risk levels ($α$), validating APSS as an effective uncertainty metric. This work establishes a principled statistical framework for trustworthy LLM deployment in high-stakes QA applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
