Predicting LLM Correctness in Prosthodontics Using Metadata and Hallucination Signals
1 Department of Data Science, Monash University Indonesia, Tangerang, Indonesia;2 Independent Researcher;3 Department of Prosthodontics, University of Pittsburgh, Pittsburgh, Pennsylvania;4 Department of Restorative Sciences, University of North Carolina Adams School of Dentistry, Chapel Hill, North Carolina;5 Department of Computer Science, Boston University, Boston, Massachusetts
Lucky Susanto, Anasta Pranawijayana, Cortino Sukotjo, Soni Prasad, Derry Wijaya
Large language models (LLMs) are increasingly adopted in high-stakes domains such as healthcare and medical education, where the risk of generating factually incorrect (i.e., hallucinated) information is a major concern. While significant efforts have been made to detect and mitigate such hallucinations, predicting whether an LLM's response is correct remains a critical yet underexplored problem. This study investigates the feasibility of predicting correctness by analyzing a general-purpose model (GPT-4o) and a reasoning-centric model (OSS-120B) on a multiple-choice prosthodontics exam. We utilize metadata and hallucination signals across three distinct prompting strategies to build a correctness predictor for each (model, prompting) pair. Our findings demonstrate that this metadata-based approach can improve accuracy by up to +7.14% and achieve a precision of 83.12% over a baseline that assumes all answers are correct. We further show that while actual hallucination is a strong indicator of incorrectness, metadata signals alone are not reliable predictors of hallucination. Finally, we reveal that prompting strategies, despite not affecting overall accuracy, significantly alter the models' internal behaviors and the predictive utility of their metadata. These results present a promising direction for developing reliability signals in LLMs but also highlight that the methods explored in this paper are not yet robust enough for critical, high-stakes deployment.
Hidden in the Metadata: Stealth Poisoning Attacks on Multimodal Retrieval-Augmented Generation
The City University of New York, CSI, Staten Island, NY 10314, USA(纽约城市大学,CSI,史泰登岛分校);The City University of New York, Graduate Center, New York, NY 10016, USA(纽约城市大学研究生中心)
VehAnchor: Metadata-Free Metric Scale Recovery from Vehicle Cues in Aerial Imagery
Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院航空航天信息研究所);Tandon School of Engineering, New York University(纽约大学工学院);School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences(中国科学院大学电子电气与通信工程学院)
AgriRegion: Region-Aware Retrieval for High-Fidelity Agricultural Advice
North Carolina A\&T State University(北卡罗来纳A&T州立大学);The University of Alabama(阿拉巴马大学);North Carolina Agricultural and Technical State University(北卡罗来纳农业与技术州立大学);Amazon AWS(亚马逊AWS)
DermaSynth: Rich Synthetic Image-Text Pairs Using Open Access Dermatology Datasets
Imperial College London(帝国理工学院);Yildiz Technical University(伊尔德兹技术大学);Usak Research and Training Hospital(乌萨克研究与培训医院);Istanbul Medeniyet University(伊斯坦布尔梅德尼耶特大学);Universität der Bundeswehr München(慕尼黑联邦国防军大学);Istanbul Medicana Atakoy Hospital(伊斯坦布尔 Medicana Atakoy 医院)