Objective: Question answering (QA) systems have the potential to improve the quality of clinical care by providing health professionals with the latest and most relevant evidence. However, QA systems have not been widely adopted. This systematic review aims to characterize current medical QA systems, assess their suitability for healthcare, and identify areas of improvement.
Materials and methods: We searched PubMed, IEEE Xplore, ACM Digital Library, ACL Anthology and forward and backward citations on 7th February 2023. We included peer-reviewed journal and conference papers describing the design and evaluation of biomedical QA systems. Two reviewers screened titles, abstracts, and full-text articles. We conducted a narrative synthesis and risk of bias assessment for each study. We assessed the utility of biomedical QA systems.
Results: We included 79 studies and identified themes, including question realism, answer reliability, answer utility, clinical specialism, systems, usability, and evaluation methods. Clinicians' questions used to train and evaluate QA systems were restricted to certain sources, types and complexity levels. No system communicated confidence levels in the answers or sources. Many studies suffered from high risks of bias and applicability concerns. Only 8 studies completely satisfied any criterion for clinical utility, and only 7 reported user evaluations. Most systems were built with limited input from clinicians.
Discussion: While machine learning methods have led to increased accuracy, most studies imperfectly reflected real-world healthcare information needs. Key research priorities include developing more realistic healthcare QA datasets and considering the reliability of answer sources, rather than merely focusing on accuracy.
LiTransProQA: an LLM-based Literary Translation evaluation metric with Professional Question Answering
Natural Language Learning Group (NLLG) School of Business Informatics and Mathematics University of Mannheim（曼海姆大学商业信息学与数学学院自然语言学习组）；University of Aberdeen, Department of Computing Science（阿伯丁大学计算机科学系）；University of Gent, Department of Translation, Interpreting and Communication（根特大学翻译、口译与传播系）；University of Technology Nuremberg (UTN), Department Engineering（纽伦堡技术大学工程系）；Natural Language Learning and Generation (NLLG) Lab（自然语言学习与生成（NLLG）实验室）
ProSwitch: Knowledge-Guided Instruction Tuning to Switch Between Professional and Non-Professional Responses
Zhejiang University of Science and Technology（浙江科技大学）；Fudan University（复旦大学）；Zhejiang University（浙江大学）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）
JMLR: Joint Medical LLM and Retrieval Training for Enhancing Reasoning and Professional Question Answering Capability
Manning College of Information and Computer Sciences, University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校信息与计算机科学学院）；Department of Medicine, University of Massachusetts Medical School（马萨诸塞大学医学院医学系）；Miner School of Computer and Information Sciences, University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校计算机与信息科学学院）；Center for Healthcare Organization and Implementation Research, VA Bedford Health Care（弗吉尼亚州贝德福德医疗保健系统医疗保健组织与实施研究中心）
Coarse-to-Careful: Seeking Semantic-related Knowledge for Open-domain Commonsense Question Answering
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
AILQA: Evaluating AI-Driven Legal Question Answering Systems for the Indian Legal System
Indian Institute of Technology（印度理工学院）；Symbiosis Law School（共生法学院）；Indian Institute of Science Education and Research（印度科学教育与研究学院）；University of Birmingham（伯明翰大学）
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Beijing Powerlaw Intelligent Technology Co., Ltd.（北京幂律智能科技有限责任公司）
Visual Question Answering in Ophthalmology: A Progressive and Practical Perspective
School of Optometry, The Hong Kong Polytechnic University（香港理工大学视光学院）；Research Centre for SHARP Vision (RCSV), The Hong Kong Polytechnic University（香港理工大学SHARP视觉研究中心）；Centre for Eye and Vision Research (CEVR)（眼科及视觉科学研究中心）
Tencent AI Lab（腾讯人工智能实验室）；ShanghaiTech University（上海科技大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
研究了股票相关问答（StockQA）问题：自动生成股票相关问题的答案，就像专业的股票分析师根据用户的请求提供股票的行动建议一样。StockQA与以前的QA任务有很大的不同，因为（1）StockQA中的答案是自然语言句子（而不是实体或值），并且由于StockQA的动态性质，几乎不可能从训练数据中以提取的方式获得合理的答案;（2）StockQA需要正确分析QA对中的关键字与股票的数字特征之间的关系。我们建议解决这个问题的内存增强的编码器-解码器架构，并集成不同的数字理解和生成机制，这是一个重要组成部分的StockQA。我们构建了一个包含超过180 K StockQA实例的大规模数据集，并在此基础上对各种技术组合进行了广泛的研究和比较。实验结果表明，一个混合的词-字符模型与单独的字符组件的数字处理，取得了最佳的性能。通过分析结果，我们发现，44.8%的答案产生的最好的模型仍然遭受通用答案的问题，这可以缓解一个简单的混合检索生成模型。
IMB: An Italian Medical Benchmark for Question Answering
University of Naples Federico II, Department of Electrical Engineering（那不勒斯费德里科二世大学电子工程系）；Consorzio Interuniversitario Nazionale per l'Informatica (CINI) - ITEM National Lab（国家信息计算联合研究中心（CINI）- ITEM国家实验室）；Northwestern University, Department of Computer Science, McCormick School of Engineering（西北大学计算机科学系，工程学院）
CARE: Counselor-Aligned Response Engine for Online Mental-Health Support
Ben-Gurion University（本· Gurion 大学）；School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；School of Informatics, University of Edinburgh Edinburgh UK（爱丁堡大学信息学院爱丁堡 英国）
