We propose a method for evaluating the quality of generated text by asking evaluators to count facts, and computing precision, recall, f-score, and accuracy from the raw counts. We believe this approach leads to a more objective and easier to reproduce evaluation. We apply this to the task of medical report summarisation, where measuring objective quality and accuracy is of paramount importance.
DeviceBERT: Applied Transfer Learning With Targeted Annotations and Vocabulary Enrichment to Identify Medical Device and Component Terminology in FDA Recall Summaries
Comparing Large Language Models and Traditional Machine Translation Tools for Translating Medical Consultation Summaries: A Pilot Study
Faculty of Information Technology, Monash University（信息技术学院，莫纳什大学）；Faculty of Medicine, Nursing and Health Sciences, Monash University（医学、护理与健康科学学院，莫纳什大学）；Supportive and Palliative Care Unit, Monash Health（支持与姑息护理单元，莫纳什健康）
MedDec: A Dataset for Extracting Medical Decisions from Discharge Summaries
Miner School of Computer & Information Sciences, University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校 Miner 计算机与信息科学学院）；Institute for Medical Engineering and Science, Massachusetts Institute of Technology（麻省理工学院医学工程与科学研究院）
Question-Driven Summarization of Answers to Consumer Health Questions
Lister Hill National Center for Biomedical Communications, U.S. National Library of Medicine, National Institutes of Health（美国国立卫生研究院美国国家医学图书馆利斯特希尔生物医学通信中心）
Paper Plain: Making Medical Research Papers Approachable to Healthcare Consumers with Natural Language Processing
University of Washington（华盛顿大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；University of California, Berkeley（加州大学伯克利分校）；University of Pennsylvania（宾夕法尼亚大学）
FactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidence
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Massachusetts Institute of Technology（麻省理工学院）；University of Duisburg-Essen（杜伊斯堡-埃森大学）；Institute for AI in Medicine, University Hospital Essen（埃森大学医院医学人工智能研究所）；Georgia Institute of Technology（佐治亚理工学院）；Northeastern University（东北大学）
EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries
KAIST（韩国科学技术院）；Seoul National University（首尔大学）；Seoul National University Bundang Hospital（首尔大学盆唐医院）；SAIHST, Sungkyunkwan University（成均馆大学）；Yonsei University College of Medicine（延世大学医学院）；Gangnam Severance Hospital（江南塞弗伦斯医院）；Severance Hospital（塞弗伦斯医院）；Seoul Medical Center（首尔医疗中心）；Seoul National University Hospital（首尔大学医院）；National Cancer Center（国立癌症中心）；Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院）；Samsung Medical Center（三星医疗中心）
A LongFormer-Based Framework for Accurate and Efficient Medical Text Summarization
Washington University in St. Louis（圣路易斯华盛顿大学）；Cornell University（康奈尔大学）；Stevens Institute of Technology（史蒂文斯理工学院）；Northeastern University（东北大学）；The Chinese University of Hong Kong（香港中文大学）；Santa Clara University（圣克拉拉大学）
Discharge Summary Hospital Course Summarisation of In Patient Electronic Health Record Text with Clinical Concept Guided Deep Pre-Trained Transformer Models
King’s College London（伦敦国王学院）；King’s College Hospital NHS Foundation Trust（国王学院医院NHS基金会信托）；University College London（伦敦大学学院）
DistillNote: Toward a Functional Evaluation Framework of LLM-Generated Clinical Note Summaries
Department of Medical Informatics, Amsterdam UMC, University of Amsterdam（医学信息学系，阿姆斯特丹大学）；Amsterdam Public Health, Methodology（阿姆斯特丹公共健康与方法学）；Institute of Mathematics and Statistics, University of São Paulo（数学与统计学研究所，圣保罗大学）
Are LLM-generated plain language summaries truly understandable? A large-scale crowdsourced evaluation
School of Information Sciences, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校信息科学学院）；Department of Radiology and Biomedical Imaging, University of California, San Francisco（加州大学旧金山分校放射科与生物医学成像系）；Eller College of Management, University of Arizona（亚利桑那大学埃勒管理学院）；Biomedical Informatics and Medical Education, University of Washington（华盛顿大学生物医学信息学与医学教育）
Phase 1 Implementation of LLM-generated Discharge Summaries showing high Adoption in a Dutch Academic Hospital
Department of Health Information Office, Data & Digitalization, University Medical Center（健康信息办公室、数据与数字化化大学医院）；Department of Trauma Surgery / Orthopedics, University Medical Center（创伤外科/骨科大学医院）；Department of Orthopaedic Surgery, Massachusetts General Hospital（骨科手术科马萨诸塞总医院）；Harvard Medical School Orthopaedic Trauma Initiative（哈佛医学院骨创伤倡议）；Department of Orthopaedic Surgery and Sports Medicine, Amsterdam UMC location University of Amsterdam（骨科手术与运动医学部门，阿姆斯特丹UMC地点大学）；Department of Orthopaedic Trauma, Flinders University Medical Centre（骨创伤部门，弗林德斯大学医疗中心）；Department of Otolaryngology – Head and Neck Surgery, University Medical Center（耳鼻喉科-头颈外科大学医院）；Department of Pediatrics, Pediatrics Nephrology, Beatrix Children’s Hospital, University Medical Center（儿科、儿科肾病学，比西特儿童医院，大学医院）
Deep Learning for Medical Text Processing: BERT Model Fine-Tuning and Comparative Study
Tulane University（杜兰大学）；Northeastern University（东北大学）；San Francisco State University（旧金山州立大学）；Brandeis University（布兰迪斯大学）；University of California, Los Angeles（加州大学洛杉矶分校）；Johns Hopkins University（约翰斯·霍普金斯大学）
ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning
Alibaba DAMO Academy（阿里巴巴达摩院）；School of Basic Medical Sciences, Lanzhou University（兰州大学基础医学学院）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学光荣人工智能学院）；Beijing Key Laboratory of Research on Large Models and Intelligent Governance（北京大型模型与智能治理研究重点实验室）；Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE（教育部下一代智能搜索与推荐工程研究中心）
Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization
Dept. of Computer Science and Engg., IIT Kharagpur（印度Kharagpur理工学院计算机科学与工程系）；Dept. of Medicine (Biomedical Informatics), Stanford University（斯坦福大学医学院（生物医学信息学））
