Language models (LMs) represent an emerging paradigm within artificial intelligence, with applications throughout the medical enterprise. A comprehensive understanding of the clinical task and awareness of the variability in performance when implemented in actual clinical environments lays the foundation for the LM application assessment. Presently, a prevalent method for evaluating the performance of these generative models relies on quantitative benchmarks. Such benchmarks have limitations and may suffer from train-to-the-test overfitting, optimizing performance for a specified test set at the cost of generalizability across other tasks and data distributions. Evaluation strategies leveraging human expertise and utilizing cost-effective computational models as evaluators are gaining interest. We discuss current state-of-the-art methodologies for assessing the performance of LM applications in healthcare and medical devices.
Generative Artificial Intelligence in Healthcare: Ethical Considerations and Assessment Checklist
Duke-NUS Medical School（杜克-新加坡国立大学医学院）；National University of Singapore（新加坡国立大学）；University of Oxford（牛津大学）；Imperial College London（帝国理工学院）；University of Antwerp（安特卫普大学）；Singapore Eye Research Institute（新加坡眼科研究所）；Singapore National Eye Centre（新加坡全国眼科中心）；Singapore Health Services（新加坡保健服务）；Singapore General Hospital（新加坡中央医院）；Hangzhou Dianzi University（杭州电子科技大学）；Nanyang Technological University（南洋理工大学）；The Chinese University of Hong Kong（香港中文大学）；Scripps Research（斯克里普斯研究所）
Can I trust my fake data -- A comprehensive quality assessment framework for synthetic tabular data in healthcare
University of Oslo（奥斯陆大学）；DNV AS（挪威船级社）；Cancer Registry of Norway（挪威癌症登记处）；Oslo University Hospital（奥斯陆大学医院）；UiT The Arctic University of Norway（挪威北极大学）
Fairboard: a quantitative framework for equity assessment of healthcare models
Queen Square Institute of Neurology, University College London（伦敦大学学院皇后广场神经科学研究所）；Dept. of Physiology, Development and Neuroscience, University of Cambridge（剑桥大学生理学、发育与神经科学系）；Laboratoire Bordelais de Recherche en Informatique (LABRI), Université de Bordeaux（波尔多大学波尔多计算机科学实验室）
Beyond Random Missingness: Clinically Rethinking for Healthcare Time Series Imputation
King’s College London（伦敦国王学院）；PyPOTS Research（PyPOTS研究中心）；University of Oxford（牛津大学）；Hong Kong University of Science and Technology（香港科技大学）；University College London（伦敦大学学院）
University of Toronto（多伦多大学）；Vector Institute（向量学院）；Adobe（奥多比）；Microsoft Research（微软研究院）
Reinforcement Learning (RL) has recently been applied to sequential estimation and prediction problems identifying and developing hypothetical treatment strategies for septic patients, with a particular focus on offline learning with observational data. In practice, successful RL relies on informative latent states derived from sequential observations to develop optimal treatment strategies. To date, how best to construct such states in a healthcare setting is an open question. In this paper, we perform an empirical study of several information encoding architectures using data from septic pat…
A Causal Framework for Mitigating Data Shifts in Healthcare
Causality in Healthcare AI Hub (CHAI)（医疗人工智能AI hub（CHAI））；The University of Edinburgh（爱丁堡大学）；University of Manchester（曼彻斯特大学）；Athena Research Center（阿塔尼亚研究中心）；NKU Athens（雅典NKU）；King’s College London（伦敦国王学院）；Imperial College London（伦敦帝国学院）；University of Exeter（埃克塞特大学）；University College London（伦敦大学学院）
Institute for Informatics and Telematics (IIT), National Research Council of Italy (CNR)（意大利国家研究委员会（CNR）信息学与电信研究所（IIT））
Nowadays Artificial Intelligence (AI) has become a fundamental component of healthcare applications, both clinical and remote, but the best performing AI systems are often too complex to be self-explaining. Explainable AI (XAI) techniques are defined to unveil the reasoning behind the system's predictions and decisions, and they become even more critical when dealing with sensitive and personal health data. It is worth noting that XAI has not gathered the same attention across different research areas and data types, especially in healthcare. In particular, many clinical and remote health appl…
University of Waterloo（滑铁卢大学）；Waterloo Artificial Intelligence Institute（滑铁卢人工智能研究所）；DarwinAI Corp.（DarwinAI公司）
The quantitative analysis of information structure through a deep neural network (DNN) can unveil new insights into the theoretical performance of DNN architectures. Two very promising avenues of research towards quantitative information structure analysis are: 1) layer similarity (LS) strategies focused on the inter-layer feature similarity, and 2) intrinsic dimensionality (ID) strategies focused on layer-wise data dimensionality using pairwise information. Inspired by both LS and ID strategies for quantitative information structure analysis, we introduce two novel complimentary methods for i…
Review of multimodal machine learning approaches in healthcare
Oxford Internet Institute, University of Oxford（牛津大学互联网研究所）；Institute for People-Centred AI, University of Surrey（萨里大学以人为本人工智能研究所）；Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London（伦敦大学学院惠康/英国工程与物理科学研究委员会介入与外科科学中心）
Automated Ensemble Multimodal Machine Learning for Healthcare
Department of Statistics, University of Oxford（统计系，牛津大学）；Division of Medical Image Computing, German Cancer Research Center (DKFZ)（医学影像计算部，德国癌症研究中心（DKFZ））；Medical Faculty Heidelberg, Heidelberg University（海德堡大学医学系）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；Pattern Analysis and Learning Group, Department of Radiation Oncology, Heidelberg University Hospital（模式分析与学习小组，海德堡大学医院放射肿瘤学系）；National Center for Tumor Diseases (NCT) Heidelberg（海德堡国家肿瘤疾病中心（NCT））；Department of Applied Mathematics and Theoretical Physics, University of Cambridge（应用数学与理论物理系，剑桥大学）
Santa Clara University（圣克拉拉大学）；University of Utah（犹他大学）
Traditional machine learning methods face two main challenges in dealing with healthcare predictive analytics tasks. First, the high-dimensional nature of healthcare data needs labor-intensive and time-consuming processes to select an appropriate set of features for each new task. Second, these methods depend on feature engineering to capture the sequential nature of patient data, which may not adequately leverage the temporal patterns of the medical events and their dependencies. Recent deep learning methods have shown promising performance for various healthcare prediction tasks by addressin…
Bidirectional human-AI collaboration in brain tumour assessments improves both expert human and AI agent performance
University College London（伦敦大学学院）；National Hospital for Neurology and Neurosurgery（国立神经病学和神经外科医院）；NVIDIA（英伟达）；Great Ormond Street Hospital for Children（大奥蒙德街儿童医院）；Royal National Orthopaedic Hospital（皇家国立骨科医院）；University College Hospitals NHS Foundation Trust（伦敦大学学院医院国民保健服务基金会信托）；Royal Free Hospital（皇家自由医院）；Imperial College Healthcare NHS Trust（帝国理工学院医疗保健国民保健服务信托）；Imperial College London（帝国理工学院）
Université Libre de Bruxelles（布鲁塞尔自由大学）；Worldline（世界线）
Credit card fraud detection is a very challenging problem because of the specific nature of transaction data and the labeling process. The transaction data is peculiar because they are obtained in a streaming fashion, they are strongly imbalanced and prone to non-stationarity. The labeling is the outcome of an active learning process, as every day human investigators contact only a small number of cardholders (associated to the riskiest transactions) and obtain the class (fraud or genuine) of the related transactions. An adequate selection of the set of cardholders is therefore crucial for an …
CNA: An AI-Oriented Comprehensive Normalized Assessment for Healthy Status and Application to Optimize RRT Strategies by Reinforcement Learning
Chongqing Institute of Green and Intelligent Technology, CAS（中国科学院重庆绿色智能技术研究院）；Chongqing School, University of Chinese Academy of Sciences（中国科学院大学重庆学院）；Department of Anesthesiology, Southwest Hospital, Third Military Medical University (Army Medical University)（陆军军医大学（第三军医大学）西南医院麻醉科）；College of Computer and Information Science, Southwest University（西南大学计算机与信息科学学院）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）
