Large language models (LLMs) are currently being used to answer medical questions across a variety of clinical domains. Recent top-performing commercial LLMs, in particular, are also capable of citing sources to support their responses. In this paper, we ask: do the sources that LLMs generate actually support the claims that they make? To answer this, we propose three contributions. First, as expert medical annotations are an expensive and time-consuming bottleneck for scalable evaluation, we demonstrate that GPT-4 is highly accurate in validating source relevance, agreeing 88% of the time with a panel of medical doctors. Second, we develop an end-to-end, automated pipeline called \textit{SourceCheckup} and use it to evaluate five top-performing LLMs on a dataset of 1200 generated questions, totaling over 40K pairs of statements and sources. Interestingly, we find that between ~50% to 90% of LLM responses are not fully supported by the sources they provide. We also evaluate GPT-4 with retrieval augmented generation (RAG) and find that, even still, around 30\% of individual statements are unsupported, while nearly half of its responses are not fully supported. Third, we open-source our curated dataset of medical questions and expert annotations for future evaluations. Given the rapid pace of LLM development and the potential harms of incorrect or outdated medical information, it is crucial to also understand and quantify their capability to produce relevant, trustworthy medical references.
CiteLLM: An Agentic Platform for Trustworthy Scientific Reference Discovery
Hong Kong Polytechnic University（香港理工大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Swiss Federal Technology Institute of Lausanne (EPFL)（洛桑联邦理工学院）；Hong Kong University of Science and Technology (HKUST)（香港科学大学）
Representation-guided in-context learning for medical image interpretation with multimodal large language models
Schepens Eye Research Institute of Massachusetts Eye and Ear（麻省眼耳医院谢本斯眼科研究所）；Harvard Medical School（哈佛医学院）；Massachusetts Eye and Ear（麻省眼耳医院）；National Taiwan University Hospital（台湾大学医院）；Massachusetts Institute of Technology（麻省理工学院）；Northwestern University（西北大学）；Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University（哈佛大学肯普纳自然与人工智能研究所）；Harvard University（哈佛大学）；Harvard Data Science Initiative, Harvard University（哈佛大学数据科学倡议）
RAU: Reference-based Anatomical Understanding with Vision Language Models
United Imaging Intelligence（联合影像智能）；School of Computing, University of Georgia（佐治亚大学计算机学院）；Department of Electrical and Computer Engineering, Northwestern University（西北大学电气与计算机工程系）
Benchmarking Clinical Decision Pathway Adherence in Large Language Models
Tongji University（同济大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Tongji University School of Medicine（同济大学医学院）；Sun Yat-sen University（中山大学）；Fuzhou University（福州大学）；Shenzhen Loop Area Institute（深圳河套学院）
STER-VLM: Spatio-Temporal With Enhanced Reference Vision-Language Models
Ho Chi Minh University of Technology（胡志明理工大学）；Vietnamese-German University（越德大学）；Ho Chi Minh University of Science（胡志明理工大学）；University of Information Technology（信息科技大学）；Vietnam National University, Ho Chi Minh city（越南国家大学，胡志明市）
MedICaT: A Dataset of Medical Images, Captions, and Textual References
University of Washington（华盛顿大学）；Tel Aviv University（特拉维夫大学）；Swedish Medical Group（瑞典医疗集团）；University of California, Irvine（加州大学欧文分校）；Allen Institute for AI（艾伦人工智能研究所）
