This paper introduces MedExQA, a novel benchmark in medical question-answering, to evaluate large language models' (LLMs) understanding of medical knowledge through explanations. By constructing datasets across five distinct medical specialties that are underrepresented in current datasets and further incorporating multiple explanations for each question-answer pair, we address a major gap in current medical QA benchmarks which is the absence of comprehensive assessments of LLMs' ability to generate nuanced medical explanations. Our work highlights the importance of explainability in medical LLMs, proposes an effective methodology for evaluating models beyond classification accuracy, and sheds light on one specific domain, speech language pathology, where current LLMs including GPT4 lack good understanding. Our results show generation evaluation with multiple explanations aligns better with human assessment, highlighting an opportunity for a more robust automated comprehension assessment for LLMs. To diversify open-source medical LLMs (currently mostly based on Llama2), this work also proposes a new medical model, MedPhi-2, based on Phi-2 (2.7B). The model outperformed medical LLMs based on Llama2-70B in generating explanations, showing its effectiveness in the resource-constrained medical domain. We will share our benchmark datasets and the trained model.
MediQAl: A French Medical Question Answering Dataset for Knowledge and Reasoning Evaluation
Data Clinic, University Hospital of Nantes, France（南特大学医院数据诊所，法国）；Nantes Université, École Centrale Nantes, CNRS, LS2N, France（南特大学，中央理工学院南特分校，国家科学研究中心，LS2N，法国）
Institute of Science and Technology, Taikang Insurance Group（泰康保险集团科学技术研究院）
In this paper, we introduce MeDiaQA, a novel question answering(QA) dataset, which constructed on real online Medical Dialogues. It contains 22k multiple-choice questions annotated by human for over 11k dialogues with 120k utterances between patients and doctors, covering 150 specialties of diseases, which are collected from haodf.com and dxy.com. MeDiaQA is the first QA dataset where reasoning over medical dialogues, especially their quantitative contents. The dataset has the potential to test the computing, reasoning and understanding ability of models across multi-turn dialogues, which is c…
PerMedCQA: Benchmarking Large Language Models on Medical Consumer Question Answering in Persian Language
School of Computer Science Institute for Research in Fundamental Sciences（计算机科学学院基础科学研究研究所）；School of Electrical and Computer Engineering University of Tehran（电气与计算机工程学院塔里班大学）；Department of Computer Science Faculty of Mathematical Sciences Alzahra University（计算机科学系数学科学学院阿扎赫拉大学）
Computer Science and Artificial Intelligence, MIT（麻省理工学院计算机科学与人工智能系）；Tongji Medical College, HUST（华中科技大学同济医学院）
Open domain question answering (OpenQA) tasks have been recently attracting more and more attention from the natural language processing (NLP) community. In this work, we present the first free-form multiple-choice OpenQA dataset for solving medical problems, MedQA, collected from the professional medical board exams. It covers three languages: English, simplified Chinese, and traditional Chinese, and contains 12,723, 34,251, and 14,123 questions for the three languages, respectively. We implement both rule-based and popular neural methods by sequentially combining a document retriever and a m…
RGAR: Recurrence Generation-augmented Retrieval for Factual-aware Medical Question Answering
King’s College London（伦敦国王学院）；School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）；School of Electrical and Computer Engineering, Carnegie Mellon University（卡内基梅隆大学电气与计算机工程学院）
Tencent Medical AI Lab（腾讯医疗人工智能实验室）；State University of New York at Buffalo（纽约州立大学布法罗分校）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；JD Urban Computing Business Unit（京东城市计算事业部）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
The past few years have witnessed the flourishing of crowdsourced medical question answering (Q&A) websites. Patients who have medical information demands tend to post questions about their health conditions on these crowdsourced Q&A websites and get answers from other users. However, we observe that a large portion of new medical questions cannot be answered in time or receive only few answers from these websites. On the other hand, we notice that solved questions have great potential to solve this challenge. Motivated by these, we propose an end-to-end system that can automatically find simi…
Despite the great success of word embedding, sentence embedding remains a not-well-solved problem. In this paper, we present a supervised learning framework to exploit sentence embedding for the medical question answering task. The learning framework consists of two main parts: 1) a sentence embedding producing module, and 2) a scoring module. The former is developed with contextual self-attention and multi-scale techniques to encode a sentence into an embedding tensor. This module is shortly called Contextual self-Attention Multi-scale Sentence Embedding (CAMSE). The latter employs two scorin…
This paper introduces MedMCQA, a new large-scale, Multiple-Choice Question Answering (MCQA) dataset designed to address real-world medical entrance exam questions. More than 194k high-quality AIIMS \& NEET PG entrance exam MCQs covering 2.4k healthcare topics and 21 medical subjects are collected with an average token length of 12.77 and high topical diversity. Each sample contains a question, correct answer(s), and other options which requires a deeper language understanding as it tests the 10+ reasoning abilities of a model across a wide range of medical subjects \& topics. A detailed explan…
The Hong Kong Polytechnic University（香港理工大学）；West China Hospital of Sichuan University（四川大学华西医院）；Sichuan Academy of Medical Sciences（四川省医学科学院）；Sichuan Provincial People's Hospital（四川省人民医院）
Medical visual question answering (Med-VQA) has tremendous potential in healthcare. However, the development of this technology is hindered by the lacking of publicly-available and high-quality labeled datasets for training and evaluation. In this paper, we present a large bilingual dataset, SLAKE, with comprehensive semantic labels annotated by experienced physicians and a new structural medical knowledge base for Med-VQA. Besides, SLAKE includes richer modalities and covers more human body parts than the currently available dataset. We show that SLAKE can be used to facilitate the developmen…
Exploring the Role of Knowledge Graph-Based RAG in Japanese Medical Question Answering with Small-Scale LLMs
The University of Tokyo（东京大学）；NEC Laboratories America（NEC美国实验室）；Dokkyo Medical University（多摩医学大学）；Graduate School of Frontier Sciences, The University of Tokyo（东京大学前沿科学研究生院）
A Systematic Study of Retrieval Pipeline Design for Retrieval-Augmented Medical Question Answering
Department of Mechatronics & Industrial Engineering, Chittagong University of Engineering & Technology（吉大港工程与技术大学机电与工业工程系）；Department of Mechanical Engineering, Chittagong University of Engineering & Technology（吉大港工程与技术大学机械工程系）
University of California San Diego（加州大学圣迭戈分校）；Beijing University of Technology（北京工业大学）；Carnegie Mellon University（卡内基梅隆大学）
Is it possible to develop an "AI Pathologist" to pass the board-certified examination of the American Board of Pathology? To achieve this goal, the first step is to create a visual question answering (VQA) dataset where the AI agent is presented with a pathology image together with a question and is asked to give the correct answer. Our work makes the first attempt to build such a dataset. Different from creating general-domain VQA datasets where the images are widely accessible and there are many crowdsourcing workers available and capable of generating question-answer pairs, developing a med…
Optimizing Medical Question-Answering Systems: A Comparative Study of Fine-Tuned and Zero-Shot Large Language Models with RAG Framework
Department of Electrical Engineering；Computer Science University of Toledo Toledo, USA；Institute of Mathematical Sciences Claremont Graduate University Claremont, USA；Department of Bioengineering University of Toledo Toledo, USA；Department of Computer Science Bowling Green State University Bowling Green, USA
Improving Reliability and Explainability of Medical Question Answering through Atomic Fact Checking in Retrieval-Augmented LLMs
Technical University of Munich（慕尼黑工业大学）；TUM University Hospital Rechts der Isar（慕尼黑工业大学 Rechts der Isar 大学医院）；German Heart Center Munich（慕尼黑德国心脏中心）；Helmholtz Zentrum München（慕尼黑亥姆霍兹中心）；German Consortium for Translational Cancer Research（德国转化癌症研究联盟）
Integrating UMLS Knowledge into Large Language Models for Medical Question Answering
National University of Singapore（新加坡国立大学）；Duke-NUS Medical School（杜克-新加坡国立大学医学院）；University of Tokyo（东京大学）；National Institute of Advanced Industrial Science and Technology（日本产业技术综合研究所）；Singapore General Hospital（新加坡中央医院）；Zhejiang Lab（之江实验室）；National Institutes of Health（美国国立卫生研究院）；Yale University（耶鲁大学）
Recent advances in Natural Language Processing (NLP), and specifically automated Question Answering (QA) systems, have demonstrated both impressive linguistic fluency and a pernicious tendency to reflect social biases. In this study, we introduce Q-Pain, a dataset for assessing bias in medical QA in the context of pain management, one of the most challenging forms of clinical decision-making. Along with the dataset, we propose a new, rigorous framework, including a sample experimental design, to measure the potential biases present when making treatment decisions. We demonstrate its use by ass…
IMB: An Italian Medical Benchmark for Question Answering
University of Naples Federico II, Department of Electrical Engineering（那不勒斯费德里科二世大学电子工程系）；Consorzio Interuniversitario Nazionale per l'Informatica (CINI) - ITEM National Lab（国家信息计算联合研究中心（CINI）- ITEM国家实验室）；Northwestern University, Department of Computer Science, McCormick School of Engineering（西北大学计算机科学系，工程学院）
Our research is in the relatively unexplored area of question answering technologies for patient-specific questions over their electronic health records. A large dataset of human expert curated question and answer pairs is an important pre-requisite for developing, training and evaluating any question answering system that is powered by machine learning. In this paper, we describe a process for creating such a dataset of questions and answers. Our methodology is replicable, can be conducted by medical students as annotators, and results in high inter-annotator agreement (0.71 Cohen's kappa). O…
Medical multiple-choice question answering (MCQA) is particularly difficult. Questions may describe patient symptoms and ask for the correct diagnosis, which requires domain knowledge and complex reasoning. Standard language modeling pretraining alone is not sufficient to achieve the best results. \citet{jin2020disease} showed that focusing masked language modeling on disease name prediction when using medical encyclopedic paragraphs as input leads to considerable MCQA accuracy improvement. In this work, we show that (1) fine-tuning on generated MCQA dataset outperforms the masked language mod…
University of California, San Diego（加州大学圣地亚哥分校）；Adobe Research（Adobe研究院）
Current medical question answering systems have difficulty processing long, detailed and informally worded questions submitted by patients, called Consumer Health Questions (CHQs). To address this issue, we introduce a medical question understanding and answering system with knowledge grounding and semantic self-supervision. Our system is a pipeline that first summarizes a long, medical, user-written question, using a supervised summarization loss. Then, our system performs a two-step retrieval to return answers. The system first matches the summarized user question with an FAQ from a trusted …
Medical image visual question answering (VQA) is a task to answer clinical questions, given a radiographic image, which is a challenging problem that requires a model to integrate both vision and language information. To solve medical VQA problems with a limited number of training data, pretrain-finetune paradigm is widely used to improve the model generalization. In this paper, we propose a self-supervised method that applies Masked image modeling, Masked language modeling, Image text matching and Image text alignment via contrastive learning (M2I2) for pretraining on medical image caption da…
University of Information Technology（信息技术大学）；Vietnam National University Ho Chi Minh City（越南胡志明市国家大学）
Question answering (QA) systems have gained explosive attention in recent years. However, QA tasks in Vietnamese do not have many datasets. Significantly, there is mostly no dataset in the medical domain. Therefore, we built a Vietnamese Healthcare Question Answering dataset (ViHealthQA), including 10,015 question-answer passage pairs for this task, in which questions from health-interested users were asked on prestigious health websites and answers from highly qualified experts. This paper proposes a two-stage QA system based on Sentence-BERT (SBERT) using multiple negatives ranking (MNR) los…
Correctness Coverage Evaluation for Medical Multiple-Choice Question Answering Based on the Enhanced Conformal Prediction Framework
School of Electronic and Information Engineering, Tongji University（同济大学电子与信息工程学院）；School of Electrical and Information Engineering, Hunan University（湖南大学电气与信息工程学院）；School of Economics and Management, Fuzhou University（福州市经济学与管理学院）
CURE: Confidence-driven Unified Reasoning Ensemble Framework for Medical Question Answering
School of Information Technology and Computer Science, Nile University（信息科技与计算机科学学院，尼罗大学）；Graduate School of Information Science, University of Hyogo（信息科学研究生院，播磨大学）；Advanced Medical Engineering Research Institute, University of Hyogo（先进医疗工程研究所，播磨大学）
Adaptive Memory and Reflection Multi-Agent System for Medical Question Answering
School of Computing and Mathematics, Manchester Metropolitan University（曼彻斯特城市大学计算与数学学院）；Electrical and Computer Engineering, Technion – Israel Institute of Technology（以色列理工学院电气与计算机工程系）；School of Science and Technology, Hong Kong Metropolitan University（香港都会大学科学与科技学院）
