Benchmarking Automated Clinical Language Simplification: Dataset, Algorithm, and Evaluation
College of Information Sciences and Technology, Pennsylvania State University（宾夕法尼亚州立大学信息科学与技术学院）；School of Software Technology, Dalian University of Technology（大连理工大学软件学院）；Relativity（相对论公司）
Junyu Luo, Zifei Zheng, Hanzhong Ye, Muchao Ye, Yaqing Wang, Quanzeng You, Cao Xiao, Fenglong Ma
Patients with low health literacy usually have difficulty understanding medical jargon and the complex structure of professional medical language. Although some studies are proposed to automatically translate expert language into layperson-understandable language, only a few of them focus on both accuracy and readability aspects simultaneously in the clinical domain. Thus, simplification of the clinical language is still a challenging task, but unfortunately, it is not yet fully addressed in previous work. To benchmark this task, we construct a new dataset named MedLane to support the development and evaluation of automated clinical language simplification approaches. Besides, we propose a new model called DECLARE that follows the human annotation procedure and achieves state-of-the-art performance compared with eight strong baselines. To fairly evaluate the performance, we also propose three specific evaluation metrics. Experimental results demonstrate the utility of the annotated MedLane dataset and the effectiveness of the proposed model DECLARE.
Massachusetts Institute of Technology（麻省理工学院）；Massachusetts General Hospital（麻省总医院）
Clinical studies often require understanding elements of a patient's narrative that exist only in free text clinical notes. To transform notes into structured data for downstream use, these elements are commonly extracted and normalized to medical vocabularies. In this work, we audit the performance of and indicate areas of improvement for state-of-the-art systems. We find that high task accuracies for clinical entity normalization systems on the 2019 n2c2 Shared Task are misleading, and underlying performance is still brittle. Normalization accuracy is high for common concepts (95.3%), but mu…
Clinical notes are an efficient way to record patient information but are notoriously hard to decipher for non-experts. Automatically simplifying medical text can empower patients with valuable information about their health, while saving clinicians time. We present a novel approach to automated simplification of medical text based on word frequencies and language modelling, grounded on medical ontologies enriched with layman terms. We release a new dataset of pairs of publicly available medical sentences and a version of them simplified by clinicians. Also, we define a novel text simplificati…
Benchmarking Deep Learning-Based Reconstruction Methods for Photoacoustic Computed Tomography with Clinically Relevant Synthetic Datasets
Department of Bioengineering, University of Illinois Urbana-Champaign（生物工程系，伊利诺伊大学厄巴纳-香槟分校）；Department of Electrical and Computer Engineering, University of Illinois Urbana-Champaign（电气与计算机工程系，伊利诺伊大学厄巴纳-香槟分校）；Department of Biomedical Engineering, The University of Texas at Austin（生物医学工程系，德克萨斯大学奥斯汀分校）；Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin（奥登计算工程与科学研究所，德克萨斯大学奥斯汀分校）
Dental-TriageBench: Benchmarking Multimodal Reasoning for Hierarchical Dental Triage
School of Computing and Data Science, The University of Hong Kong（香港大学计算与数据科学学院）；Faculty of Dentistry, The University of Hong Kong（香港大学牙科学院）；The Prince Philip Dental Hospital（菲利普王子牙科医院）；Li Ka Shing Faculty of Medicine, The University of Hong Kong（香港大学利滋医学学院）；The Hong Kong University of Science and Technology（香港科学与技术大学）
National Center for Biotechnology Information（国家生物技术信息中心）；National Library of Medicine, National Institutes of Health（美国国立卫生研究院国家医学图书馆）
Inspired by the success of the General Language Understanding Evaluation benchmark, we introduce the Biomedical Language Understanding Evaluation (BLUE) benchmark to facilitate research in the development of pre-training language representations in the biomedicine domain. The benchmark consists of five tasks with ten datasets that cover both biomedical and clinical texts with different dataset sizes and difficulties. We also evaluate several baselines based on BERT and ELMo and find that the BERT model pre-trained on PubMed abstracts and MIMIC-III clinical notes achieves the best results. We m…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Northeastern University（东北大学）；King’s College London（伦敦国王学院）
We consider the problem of learning to simplify medical texts. This is important because most reliable, up-to-date information in biomedicine is dense with jargon and thus practically inaccessible to the lay audience. Furthermore, manual simplification does not scale to the rapidly growing body of biomedical literature, motivating the need for automated approaches. Unfortunately, there are no large-scale resources available for this task. In this work we introduce a new corpus of parallel texts in English comprising technical and lay summaries of all published evidence pertaining to different …
Benchmarking the Thinking Mode of Multimodal Large Language Models in Clinical Tasks
Bytedance（字节跳动）；Peking University（北京大学）；The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；Mohamed bin Zayed University of Artificial Intelligence（马尔代夫人工智能大学）；Stanford University（斯坦福大学）；University of Michigan（密歇根大学）
MedSNIP: Building and Benchmarking Snippet-Level Granularity for Medical Fact Verification
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Yale University（耶鲁大学）；Korea University（高丽大学）；Beijing Normal University（北京师范大学）；INSAIT, Sofia University, “St. Kliment Ohridski”（索非亚大学INSAIT研究所）
AMPLIFAI: A Multiphase CT Dataset for Benchmarking Clinical Reasoning in LI-RADS Assessment of Liver Lesions
University of Maryland, College Park（马里兰大学帕克分校）；University of Maryland School of Medicine（马里兰大学医学院）；University of Maryland Institute for Health Computing（马里兰大学健康计算研究所）；University of Maryland Medical System（马里兰大学医学系统）
Redefining Simplicity: Benchmarking Large Language Models from Lexical to Document Simplification
School of Information and Engineering, Yangzhou University（扬州大学信息工程学院）；School of Computer Science and Information Technology, Hefei University of Technology（合肥工业大学计算机与信息学院）
Benchmarking and Enhancing PPG-Based Cuffless Blood Pressure Estimation Methods
Department of Engineering Technology, University of Houston, Texas, USA（工程科技系，休斯顿大学，德克萨斯州，美国）；Department of Electrical and Computer Engineering, University of Houston, Texas, USA（电气与计算机工程系，休斯顿大学，德克萨斯州，美国）；Department of Information Science Technology, University of Houston, Texas, USA（信息科学科技系，休斯顿大学，德克萨斯州，美国）；Department of Cardiovascular Surgery, Houston Methodist Hospital, Texas, USA（心血管外科系，休斯顿方法医院，德克萨斯州，美国）；Department of Biomedical Engineering, University of Houston, Texas, USA（生物医学工程系，休斯顿大学，德克萨斯州，美国）
A Comprehensive Benchmark for COVID-19 Predictive Modeling Using Electronic Health Records in Intensive Care
University of Edinburgh（爱丁堡大学）；Health Data Research UK（英国健康数据研究中心）；Peking University（北京大学）；Peking University Third Hospital（北京大学第三医院）；Kaihong Digital（凯宏数字）
ALPEC: A Comprehensive Evaluation Framework and Dataset for Machine Learning-Based Arousal Detection in Clinical Practice
University of Tübingen（蒂宾根大学）；IT-Designers Gruppe（IT-Designers集团）；Aalen University of Applied Sciences（阿伦应用技术大学）；Klinikum Esslingen（埃斯林根医院）；Technical University of Munich（慕尼黑工业大学）
U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding
Fudan University（复旦大学）；University of Oxford（牛津大学）；Dolphin AI；Beihang University（北京航空航天大学）；Hong Kong Baptist University（香港浸会大学）；University of Cambridge（剑桥大学）；The Chinese University of Hong Kong（香港中文大学）；Esonic Imaging；Technical University of Denmark（丹麦技术大学）
Benchmarking emergency department triage prediction models with machine learning and large public electronic health records
Duke-NUS Medical School（杜克-新加坡国立大学医学院）；Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局高性能计算研究所）；Nanyang Technological University（南洋理工大学）；Monash University（莫纳什大学）；National University of Singapore（新加坡国立大学）；Duke University（杜克大学）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Harvard Medical School（哈佛医学院）；Massachusetts Institute of Technology（麻省理工学院）；Singapore General Hospital（新加坡中央医院）；Singapore Health Services（新加坡保健服务集团）
Clinical deployment of deep learning algorithms for chest x-ray interpretation requires a solution that can integrate into the vast spectrum of clinical workflows across the world. An appealing approach to scaled deployment is to leverage the ubiquity of smartphones by capturing photos of x-rays to share with clinicians using messaging services like WhatsApp. However, the application of chest x-ray algorithms to photos of chest x-rays requires reliable classification in the presence of artifacts not typically encountered in digital x-rays used to train machine learning models. We introduce Che…
Benchmarking Machine Learning Architectures for Antimicrobial Stewardship in Pediatric ICUs
Department of Intensive Care and Neonatology and Children’s Research Center University of Zurich University Children’s Hospital Zurich（重症护理与新生儿科及儿童研究中心，苏黎世大学苏黎世儿童医院）；Department of Health Sciences and Technology ETH Zurich（健康科学与技术系，苏黎世联邦理工学院）；Department of Computer Science ETH Zurich（计算机科学系，苏黎世联邦理工学院）
A reproducible and extensible framework for benchmarking competing risks survival models
Cancer Research UK Scotland Centre（英国癌症研究苏格兰中心）；Institute of Genetics and Cancer（遗传与癌症研究所）；University of Edinburgh（爱丁堡大学）；University of Augsburg（奥格斯堡大学）
Benchmarking Pathology Foundation Models for Breast Cancer Survival Prediction
Department of Medical Epidemiology and Biostatistics, Karolinska Institutet（卡罗林斯卡研究所医学流行病学与生物统计学系）；Department of Engineering Science, University of Oxford（牛津大学工程科学系）；Division of Oncology, Department of Clinical Sciences Lund, Lund University（卢德大学临床科学系肿瘤学系）；Oxford Suzhou Centre for Advanced Research, University of Oxford（牛津大学苏州先进研究中心）
MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks
National Engineering Research Center for Software Engineering, Peking University（北京大学国家软件工程研究中心）；School of Computing and Data Science, The University of Hong Kong（香港大学计算与数据科学学院）；Centre for Medical Informatics, The University of Edinburgh（爱丁堡大学医学信息学中心）；ETH Zurich（苏黎世联邦理工学院）；Health Data Research UK（英国健康数据研究所）；Key Laboratory of High Confidence Software Technologies, Ministry of Education（教育部高可信软件技术重点实验室）
Machine-learning for photoplethysmography analysis: Benchmarking feature, image, and signal-based approaches
Carl von Ossietzky Universität Oldenburg（奥尔登堡卡尔·冯·奥西特齐大学）；Laboratoire national de métrologie et d’essais（国家计量与测试实验室）；Biomedical Engineering Institute, Kaunas University of Technology（克莱萨斯理工大学生物医学工程研究所）；Physikalisch-Technische Bundesanstalt（德国物理技术部）；National Physical Laboratory（国家物理实验室）；King’s College London（伦敦国王学院）；University of Surrey（萨里大学）；Department of Public Health and Primary Care, University of Cambridge（剑桥大学公共卫生与初级保健系）
Paradigm（第四范式）；Universitat de Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）；LISN/CNRS/INRIA, Univ. Paris-Saclay（巴黎萨克雷大学LISN实验室/法国国家科学研究中心/法国国家信息与自动化研究所）；ChaLearn（ChaLearn挑战赛组织）；Univ. Grenoble Alpes/CNRS（格勒诺布尔阿尔卑斯大学/法国国家科学研究中心）；Tsinghua University（清华大学）
Obtaining standardized crowdsourced benchmark of computational methods is a major issue in data science communities. Dedicated frameworks enabling fair benchmarking in a unified environment are yet to be developed. Here we introduce Codabench, an open-source, community-driven platform for benchmarking algorithms or software agents versus datasets or tasks. A public instance of Codabench (https://www.codabench.org/) is open to everyone, free of charge, and allows benchmark organizers to compare fairly submissions, under the same setting (software, hardware, data, algorithms), with custom protoc…
University of Southern California（南加州大学）；Tsinghua University（清华大学）
Deep learning models (aka Deep Neural Networks) have revolutionized many fields including computer vision, natural language processing, speech recognition, and is being increasingly used in clinical healthcare applications. However, few works exist which have benchmarked the performance of the deep learning models with respect to the state-of-the-art machine learning models and prognostic scoring systems on publicly available healthcare datasets. In this paper, we present the benchmarking results for several clinical prediction tasks such as mortality prediction, length of stay prediction, and…
BLESS: Benchmarking Large Language Models on Sentence Simplification
University of Zurich（苏黎世大学）；National Tsing Hua University（国立清华大学）；Idiap Research Institute（Idiap研究所）；University of Manchester（曼彻斯特大学）；University of Maryland（马里兰大学）；Cohere（Cohere公司）；Cardiff University（卡迪夫大学）；Manchester Metropolitan University（曼彻斯特城市大学）
Benchmarking Foundation Models with Multimodal Public Electronic Health Records
Centre for Quantitative Medicine and Duke-NUS AI + Medical Science Initiative, Duke-NUS Medical School（定量医学中心和杜克-国立新加坡大学AI+医学科学计划，杜克-国立新加坡大学医学院）；Graduate School of Engineering, The University of Tokyo（东京大学工程研究生院）；Department of Population Health Sciences, Weill Cornell Medicine（流行病学与公共卫生科学系，韦尔·科恩医学中心）；Department of Surgery, Duke University School of Medicine（外科医学系，杜克大学医学学院）；Centre for Quantitative Medicine, Duke-NUS AI + Medical Science Initiative and Programme in Health Services and Systems Research, Duke-NUS Medical School and NUS Artificial Intelligence Institute, National University of Singapore（定量医学中心和杜克-国立新加坡大学AI+医学科学计划及健康服务与系统研究计划，杜克-国立新加坡大学医学院和新加坡国立大学人工智能研究所）
OpenECG: Benchmarking ECG Foundation Models with Public 1.2 Million Records
The First Affiliated Hospital, Jiangxi Medical College, Nanchang University（南昌大学江西医学院第一附属医院）；Nanchang University（南昌大学）；Georgia Institution of Technology（佐治亚理工学院）
University of Cambridge（剑桥大学）；University of Oxford（牛津大学）；UCLA（加州大学洛杉矶分校）
Estimating personalized effects of treatments is a complex, yet pervasive problem. To tackle it, recent developments in the machine learning (ML) literature on heterogeneous treatment effect estimation gave rise to many sophisticated, but opaque, tools: due to their flexibility, modularity and ability to learn constrained representations, neural networks in particular have become central to this literature. Unfortunately, the assets of such black boxes come at a cost: models typically involve countless nontrivial operations, making it difficult to understand what they have learned. Yet, unders…
Reproducible Benchmarking for Lung Nodule Detection and Malignancy Classification Across Multiple Low-Dose CT Datasets
Duke University School of Medicine（杜克大学医学院）；Pratt School of Engineering, Duke University（杜克大学普拉特工程学院）；University of Colorado（科罗拉多大学）；Duke University（杜克大学）
University of Minnesota（明尼苏达大学）；Carleton College（卡尔顿学院）；University of North Dakota（北达科他大学）
The morphometry of a kidney tumor revealed by contrast-enhanced Computed Tomography (CT) imaging is an important factor in clinical decision making surrounding the lesion's diagnosis and treatment. Quantitative study of the relationship between kidney tumor morphology and clinical outcomes is difficult due to data scarcity and the laborious nature of manually quantifying imaging predictors. Automatic semantic segmentation of kidneys and kidney tumors is a promising tool towards automatically quantifying a wide array of morphometric features, but no sizeable annotated dataset is currently avail…
Benchmarking Clinical Decision Pathway Adherence in Large Language Models
Tongji University（同济大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Tongji University School of Medicine（同济大学医学院）；Sun Yat-sen University（中山大学）；Fuzhou University（福州大学）；Shenzhen Loop Area Institute（深圳河套学院）
Mind the Gap: Benchmarking LLM Uncertainty and Calibration with Specialty-Aware Clinical QA and Reasoning-Based Behavioural Features
Department of Medical Informatics, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, The Netherlands（医学信息学系，阿姆斯特丹大学医学中心，阿姆斯特丹大学，阿姆斯特丹，荷兰）；Amsterdam Public Health, Methodology, Amsterdam, The Netherlands（阿姆斯特丹公共健康，方法学，阿姆斯特丹，荷兰）
