The increasing use of tools and solutions based on Large Language Models (LLMs) for various tasks in the medical domain has become a prominent trend. Their use in this highly critical and sensitive domain has thus raised important questions about their robustness, especially in response to variations in input, and the reliability of the generated outputs. This study addresses these questions by constructing a textual dataset based on the ICD-10-CM code descriptions, widely used in US hospitals and containing many clinical terms, and their easily reproducible rephrasing. We then benchmarked existing embedding models, either generalist or specialized in the clinical domain, in a semantic search task where the goal was to correctly match the rephrased text to the original description. Our results showed that generalist models performed better than clinical models, suggesting that existing clinical specialized models are more sensitive to small changes in input that confuse them. The highlighted problem of specialized models may be due to the fact that they have not been trained on sufficient data, and in particular on datasets that are not diverse enough to have a reliable global language understanding, which is still necessary for accurate handling of medical documents.
Generalist Large Language Models Outperform Clinical Tools on Medical Benchmarks
NYU Langone Health(纽约大学朗格尼医学中心);The University of Texas at Austin(得克萨斯大学奥斯汀分校);Washington University School of Medicine in St. Louis(圣路易斯华盛顿大学医学院);New York University(纽约大学)
UltraMedical: Building Specialized Generalists in Biomedicine
Tsinghua University(清华大学);University of Washington(华盛顿大学);The First Affiliated Hospital of Nanchang University(南昌大学第一附属医院);Shanghai Jiao Tong University(上海交通大学)
A Perspective for Adapting Generalist AI to Specialized Medical AI Applications and Their Challenges
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Mayo Clinic Health System(梅奥诊所健康系统);Regeneron(再生元制药公司);IQVIA(艾昆纬公司);Microsoft Research(微软研究院);Teladoc Health(Teladoc医疗公司);Harvard Medical School(哈佛医学院);Carle Illinois College of Medicine(卡尔伊利诺伊医学院)
Generalist Foundation Models Are Not Clinical Enough for Hospital Operations
Courant Institute School of Mathematics, Computing, and Data Science(Courant学院数学、计算与数据科学学院);New York University(纽约大学);Department of Neurosurgery(神经外科部);Global AI Frontier Lab(全球人工智能前沿实验室);Electrical and Computer Engineering(电气与计算机工程);Tandon School of Engineering(Tandon工程学院);Grossman School of Medicine(Grossman医学院);Department of Medicine(医学部);Department of Computer Science(计算机科学部);Department of Surgery(外科部);Division of Applied AI Technologies(应用人工智能技术部);Department of Population Health(人口健康部);ETH Zurich(苏黎世联邦理工学院)
Modality-Specialized Synergizers for Interleaved Vision-Language Generalists
Virginia Tech(弗吉尼亚理工大学);Intuit AI Research(Intuit AI研究院);Meta AI;The Chinese University of Hong Kong(香港中文大学);University of California, Davis(加州大学戴维斯分校)
Specialty-Oriented Generalist Medical AI for Chest CT Screening
Rensselaer Polytechnic Institute(伦斯勒理工学院);Wake Forest University School of Medicine(维克森林大学医学院);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院)
Towards a Psychological Generalist AI: A Survey of Current Applications of Large Language Models and Future Prospects
Wuhan University(武汉大学);Sorbonne Université(索邦大学);Institut du Cerveau - Paris Brain Institute - ICM(巴黎脑研究所-ICM);CNRS(法国国家科学研究中心);Inria(法国国家信息与自动化研究所);Inserm(法国国家健康与医学研究院);AP-HP(巴黎公共援助医院);Beijing University of Technology(北京工业大学)
TAGS: A Test-Time Generalist-Specialist Framework with Retrieval-Augmented Reasoning and Verification
Mohamed bin Zayed University of Artificial Intelligence(莫莫德·本·扎耶德人工智能大学);Monash University(墨尔本大学);Xi’an Jiaotong-Liverpool University(西安交通大学利物浦大学);University of Southampton(南安普顿大学)
EyeFound: A Multimodal Generalist Foundation Model for Ophthalmic Imaging
The Hong Kong Polytechnic University(香港理工大学);Research Centre for SHARP Vision (RCSV), The Hong Kong Polytechnic University(香港理工大学夏普视觉研究中心);Swiss Federal Institute of Technology Lausanne (EPFL)(洛桑联邦理工学院);Clemson University(克莱姆森大学);Yong Loo Lin School of Medicine, National University of Singapore(新加坡国立大学杨潞龄医学院);Singapore Eye Research Institute(新加坡眼科研究所);State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University(中山大学中山眼科中心眼科学国家重点实验室);Guangdong Provincial Key Laboratory of Ophthalmology and Visual Science(广东省眼科与视觉科学重点实验室);Guangdong Provincial Clinical Research Center for Ocular Diseases(广东省眼部疾病临床医学研究中心);Centre for Eye and Vision Research (CEVR)(眼科与视觉研究中心)
A Specialized Large Language Model for Clinical Reasoning and Diagnosis in Rare Diseases
Guangzhou Women and Children's Medical Center, Guangzhou Medical University(广州医科大学附属广州市妇女儿童医疗中心);Tianjin Medical University(天津医科大学);Guangzhou National Laboratory(广州国家实验室);Children's Hospital of Fudan University(复旦大学附属儿童医院)
LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation
College of Computer Science and Artificial Intelligence, Fudan University(计算机科学与人工智能学院,复旦大学);Shanghai Key Lab of Intelligent Information Processing(上海智能信息处理重点实验室);Shanghai AI Laboratory(上海人工智能实验室);Northwestern University(西北大学)
ChemAmp: Amplified Chemistry Tools via Composable Agents
Artificial Intelligence Innovation and Incubation Institute, Fudan University(复旦大学人工智能创新与孵化院);School of Data Science, Fudan University(复旦大学数据科学学院);College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);Department of Information and Intelligence Development, Zhongshan Hospital, Fudan University(复旦大学中山医院信息与智能发展部)
Learning structures of the French clinical language:development and validation of word embedding models using 21 million clinical reports from electronic health records
Assistance Publique – Hôpitaux de Paris(巴黎公共援助医院);Sorbonne Université(索邦大学);Inserm(法国国家健康与医学研究院);Université Sorbonne Paris Nord(索邦巴黎北大学);Centre de Recherche des Cordeliers(科德利埃研究中心);INRIA(法国国家信息与自动化研究所)
ClinicRealm: Re-evaluating Large Language Models with Conventional Machine Learning for Non-Generative Clinical Prediction Tasks
School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);Centre for Medical Informatics, The University of Edinburgh(爱丁堡大学医学信息学中心);Health Data Research UK(英国健康数据研究所);School of Computer Science, Peking University(北京大学计算机学院);National Engineering Research Center for Software Engineering, Peking University(北京大学软件工程国家工程研究中心);ETH Zurich(苏黎世联邦理工学院);School of Computing and Data Science, The University of Hong Kong(香港大学计算与数据科学学院)
Technical Report: Small Language Model for Japanese Clinical and Medicine
National Cerebral and Cardiovascular Center Hospital(国立循环器病研究中心医院);Graduate School of Medicine and Faculty of Medicine, Kyoto University(京都大学大学院医学研究科及医学部)
Specialized curricula for training vision-language models in retinal image analysis
Imperial College London(伦敦帝国学院);University of Southampton(南安普顿大学);Moorfields Eye Hospital NHS Foundation Trust(摩尔菲尔兹眼科医院NHS基金会信托);Medical University of Vienna(维也纳医科大学);Technical University of Munich(慕尼黑工业大学);Institute of Molecular and Clinical Ophthalmology Basel(巴塞尔分子与临床眼科研究所);University of Basel(巴塞尔大学)