潜在置信对齐用于大语言模型自我评估
Latent Confidence Alignment for LLM Self-Assessment
AI总结：
提出基于Rasch模型的潜在置信对齐误差（LCAE）来评估LLM自我评估与潜在错误概率的一致性，并引入项目难度作为外部信号，实验表明该方法在不影响模型能力的情况下提升自我评估质量。
摘要：
大语言模型（LLM）中的置信校准通常通过比较预测置信度与观察到的准确率来评估。然而，这种方法没有对项目难度进行建模，使得难以解释差异，也难以确定模型置信度是反映真实的自我评估还是仅仅是响应生成过程的副产品。为了解决这个问题，我们采用基于Rasch模型的潜在能力框架和元认知视角，提出潜在置信对齐误差（LCAE）来衡量模型自我评估与模型能力和项目难度所隐含的潜在错误概率之间的一致性。我们进一步将项目难度作为外部信号，并结合推理机制。在包含20个模型的医学领域数据集上的实验表明，所提出的方法在不影响模型能力的情况下提高了自我评估质量，并揭示了可靠性与推理成本之间的关联。
英文摘要：
Confidence calibration in large language models (LLMs) is commonly evaluated by comparing predicted confidence with observed accuracy. However, such approaches do not model item difficulty, making it difficult to interpret discrepancies and to determine whether model confidence reflects genuine self-assessment or is merely a byproduct of the response generation process. To address this, we adopt a Rasch model-based latent ability framework and a metacognitive perspective, and propose Latent Confidence Alignment Error (LCAE) to measure the consistency between model self-assessment and the latent error probability implied by model ability and item difficulty. We further incorporate item difficulty as an external signal with a reasoning mechanism. Experiments on a medical-domain dataset with 20 models show that the proposed approach improves self-assessment quality without affecting model ability, and reveals an association between reliability and inference cost.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
