arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.22508cs.LGcs.AI
收藏

利用元数据和幻觉信号预测牙科修复学中大语言模型的正确性

Predicting LLM Correctness in Prosthodontics Using Metadata and Hallucination Signals

1 Department of Data Science, Monash University Indonesia, Tangerang, Indonesia;2 Independent Researcher;3 Department of Prosthodontics, University of Pittsburgh, Pittsburgh, Pennsylvania;4 Department of Restorative Sciences, University of North Carolina Adams School of Dentistry, Chapel Hill, North Carolina;5 Department of Computer Science, Boston University, Boston, Massachusetts

Lucky Susanto, Anasta Pranawijayana, Cortino Sukotjo, Soni Prasad, Derry Wijaya

AI总结:

本文研究通过元数据和幻觉信号预测牙科修复学中LLM的正确性,发现元数据方法可提升准确性,但需进一步改进以适应高风险应用。

摘要:

大型语言模型(LLMs)正在越来越多地应用于高风险领域,如医疗保健和医学教育,其中生成事实性错误(即幻觉)信息的风险是一个主要问题。尽管已经做出了大量努力来检测和减轻此类幻觉,但预测LLM响应是否正确仍然是一个关键但尚未充分研究的问题。本研究探讨了通过分析通用模型(GPT-4o)和以推理为中心的模型(OSS-120B)在多项选择牙科修复学考试中的可行性。我们利用元数据和幻觉信号在三种不同的提示策略下,为每(模型,提示)对构建一个正确性预测器。我们的发现表明,这种基于元数据的方法可以将准确性提高高达+7.14%,并在基准假设所有答案都正确的基线上实现83.12%的精度。我们进一步表明,尽管实际幻觉是不正确性的强指示器,但仅靠元数据信号并不能可靠地预测幻觉。最后,我们揭示了提示策略,尽管不改变总体准确性,但显著改变了模型的内部行为及其元数据的预测效用。这些结果为开发LLM的可靠性信号提供了一个有希望的方向,但也突显了本文探讨的方法尚不足以在关键、高风险的部署中使用。

英文摘要:

Large language models (LLMs) are increasingly adopted in high-stakes domains such as healthcare and medical education, where the risk of generating factually incorrect (i.e., hallucinated) information is a major concern. While significant efforts have been made to detect and mitigate such hallucinations, predicting whether an LLM's response is correct remains a critical yet underexplored problem. This study investigates the feasibility of predicting correctness by analyzing a general-purpose model (GPT-4o) and a reasoning-centric model (OSS-120B) on a multiple-choice prosthodontics exam. We utilize metadata and hallucination signals across three distinct prompting strategies to build a correctness predictor for each (model, prompting) pair. Our findings demonstrate that this metadata-based approach can improve accuracy by up to +7.14% and achieve a precision of 83.12% over a baseline that assumes all answers are correct. We further show that while actual hallucination is a strong indicator of incorrectness, metadata signals alone are not reliable predictors of hallucination. Finally, we reveal that prompting strategies, despite not affecting overall accuracy, significantly alter the models' internal behaviors and the predictive utility of their metadata. These results present a promising direction for developing reliability signals in LLMs but also highlight that the methods explored in this paper are not yet robust enough for critical, high-stakes deployment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑