arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.27702cs.CLcs.AI
收藏

超越校准:跨语言与本地场景的时序大语言模型置信度评估

Beyond Calibration: Temporal LLM Confidence Assessment Across Languages and Local Settings

University of California, Los Angeles(加利福尼亚大学洛杉矶分校)

Sahil Kale

AI总结:

本研究区分LLM的预解决知晓感(FOK)与后解决学习判断(JOL),发现后解决置信校准更强,语言翻译下迁移性好但本地知识场景下降,且幻觉响应置信度高,揭示LLM自我评估局限。

摘要:

大型语言模型(LLM)需评估自身不确定性,置信信号可用于指导检索、自适应计算分配、路由及选择性弃权(不执行)。本研究探究问题解决不同时序阶段测得的置信信号是否编码不同形式的认知信息,且在非英语场景下仍保持可靠性。受经典元记忆理论启发,本研究区分预解决阶段的预期置信(知晓感,FOK)与后解决阶段的回溯置信(学习判断,JOL)。在两种前沿LLM、多个推理领域、翻译的马拉地语基准及新构建的真实本地马拉地语事实问题基准上评估这些信号。跨场景来看,成本更低的预解决信号校准度较弱,而后解决置信虽有额外推理成本,通常提供更强的认知对齐。置信行为在直接语言翻译下表现出惊人的迁移性,英语与马拉地语间校准趋势基本保留,但在本地马拉地语知识上大幅下降,表明仅语言迁移不足以满足特定区域信息需求。最后,幻觉响应在答案生成后仍保持持续升高的置信度,凸显当前LLM自我评估机制的重要局限。

英文摘要:

Large language models are expected to estimate their own uncertainty, with confidence signals used to guide retrieval, adaptive compute allocation, routing, and selective abstention. We investigate whether confidence signals measured at different temporal stages of problem solving encode distinct forms of epistemic information and remain reliable beyond English-centric settings. Inspired by classical meta-memory theory, we distinguish anticipatory pre-solve confidence (Feeling-of-Knowing, FOK) from retrospective post-solve confidence (Judgement-of-Learning, JOL). We evaluate these signals across two frontier LLMs, multiple reasoning domains, translated Marathi benchmarks, and a newly constructed benchmark of genuinely local Marathi factual questions. Across settings, the cheaper pre-solve signal remains weakly calibrated, while post-solve confidence generally provides stronger epistemic alignment despite its additional inference cost. Confidence behaviour transfers surprisingly well under direct language translation, with calibration trends largely preserved between English and Marathi, but degrades substantially on local Marathi knowledge, suggesting that linguistic transfer alone is insufficient for region-specific information needs. Finally, hallucinated responses maintain persistently elevated confidence after answer generation, highlighting important limitations of current LLM self-assessment mechanisms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑