Following an interaction with a patient, physicians are responsible for the submission of clinical documentation, often organized as a SOAP note. A clinical note is not simply a summary of the conversation but requires the use of appropriate medical terminology. The relevant information can then be extracted and organized according to the structure of the SOAP note. In this paper we analyze two different approaches to generate the different sections of a SOAP note based on the audio recording of the conversation, and specifically examine them in terms of note consistency. The first approach generates the sections independently, while the second method generates them all together. In this work we make use of PEGASUS-X Transformer models and observe that both methods lead to similar ROUGE values (less than 1% difference) and have no difference in terms of the Factuality metric. We perform a human evaluation to measure aspects of consistency and demonstrate that LLMs like Llama2 can be used to perform the same tasks with roughly the same agreement as the human annotators. Between the Llama2 analysis and the human reviewers we observe a Cohen Kappa inter-rater reliability of 0.79, 1.00, and 0.32 for consistency of age, gender, and body part injury, respectively. With this we demonstrate the usefulness of leveraging an LLM to measure quality indicators that can be identified by humans but are not currently captured by automatic metrics. This allows scaling evaluation to larger data sets, and we find that clinical note consistency improves by generating each new section conditioned on the output of all previously generated sections.
Evaluating Large Language Model Raters for German Open-Response Clinical Questions: A Physician-Annotated Benchmark Study of Agreement, Evaluator Bias, and Abstention
University of Luebeck（吕贝克大学）；University of Tübingen（图宾根大学）；University Hospital Schleswig-Holstein（石勒苏益格-荷尔斯泰因大学医院）；University Hospital Würzburg（维尔茨堡大学医院）；Charité – Universitätsmedizin Berlin（柏林夏里特医学院）；Genie Enterprise Deutschland GmbH
Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
Department of Machine Learning, Carnegie Mellon University, Pittsburgh, PA, USA（卡内基梅隆大学机器学习系）；Department of Computer Science and Engineering, University of California San Diego, La Jolla, CA, USA（加州大学圣地亚哥分校计算机科学与工程系）；Department of Computer Science, University of Virginia, Charlottesville, VA, USA（弗吉尼亚大学计算机科学系）；Martian Research, San Francisco, California, USA（火星研究公司）；Apart Research, San Francisco, California, USA（Apart研究公司）
Can Generic LLMs Help Analyze Child-adult Interactions Involving Children with Autism in Clinical Observation?
University of Southern California（南加州大学）；Boston University（波士顿大学）；Korea University（高丽大学）；University of California, San Francisco（加利福尼亚大学旧金山分校）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Assessing the Quality of AI-Generated Clinical Notes: A Validated Evaluation of a Large Language Model Scribe
Suki AI；Santa Clara Valley Medical Center（圣克拉拉谷医疗中心）；Stanford University School of Medicine（斯坦福大学医学院）；Stanford Cardiovascular Institute（斯坦福心血管研究所）；Hippocratic AI；The Permanente Medical Group（永久医疗集团）
Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports
Nanyang Technological University（南洋理工大学）；Technical University of Munich（慕尼黑工业大学）；Alibaba（阿里巴巴）；University of Glasgow（格拉斯哥大学）；University of Massachusetts Boston（马萨诸塞大学波士顿分校）
MedArena: Comparing LLMs for Medicine-in-the-Wild Clinician Preferences
Department of Electrical Engineering, Stanford University（斯坦福大学电气工程系）；Department of Biomedical Data Science, Stanford University（斯坦福大学生物医学数据科学系）；Division of Hospital Medicine, Department of Medicine, Stanford School of Medicine（斯坦福医学院医学部住院医学科）；Department of Radiology, St. Jude Children's Research Hospital（圣 Jude 儿童研究医院放射科）；University of California, San Francisco（旧金山大学）；Department of Pathology and Laboratory Medicine, University of Wisconsin School of Medicine and Public Health（威斯康星大学医学与公共卫生学院病理学与实验室医学系）；Doximity, San Francisco, CA, USA（Doximity公司）；Department of Medicine, Division of Rheumatology and Immunology, Vanderbilt University Medical Center（范德比尔特大学医学中心医学系风湿病与免疫学科）；Department of Neurology, Charleston Area Medical Center（查尔斯顿医疗中心神经科）；Department of Translational Medicine, Scripps Research Translational Institute（斯克里普斯研究转化研究所转化医学系）；Kaiser Permanente Division of Research（凯撒医疗集团研究部）
Extracting Social Support and Social Isolation Information from Clinical Psychiatry Notes: Comparing a Rule-based NLP System and a Large Language Model
Weill Cornell Medicine（威尔康奈尔医学院）；Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院）；Mayo Clinic（梅奥诊所）；Columbia University Vagelos College of Physicians and Surgeons（哥伦比亚大学瓦格洛斯内科医生与外科医生学院）；New York State Psychiatric Institute（纽约州精神病学研究所）
Cognitive bias in LLM reasoning compromises interpretation of clinical oncology notes
Mayo Clinic College of Medicine and Science（梅奥诊所医学与科学学院）；School of Computing and AI, Arizona State University（亚利桑那州立大学计算与人工智能学院）；Mayo Clinic Comprehensive Cancer Center（梅奥诊所综合癌症中心）；Department of Radiation Oncology, Mayo Clinic（梅奥诊所放射肿瘤科）；Department of Oncology, Mayo Clinic（梅奥诊所肿瘤科）；Department of Artificial Intelligence and Informatics, Mayo Clinic（梅奥诊所人工智能与信息学系）；Dana-Farber Cancer Institute, Harvard Medical School（哈佛医学院丹娜-法伯癌症研究所）
Using LLM-as-a-Judge/Jury to Advance Scalable, Clinically-Validated Safety Evaluations of Model Responses to Users Demonstrating Psychosis
Apart Research；Odyssean Institute；London School of Economics and Political Science（伦敦政治经济学院）；Stanford Institute for Human-Centered AI（斯坦福大学以人为本人工智能研究所）
Large Language Models as Unified Multimodal Learners for Clinical Prediction
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；Charité Universitätsmedizin Berlin（柏林夏里特大学医学中心）；DNC Information Management GmbH（DNC信息管理有限公司）；Klinik für Akut- und Notfallmedizin, Asklepios Klinikum Harburg（哈堡阿斯克勒庇俄斯医院急性与急诊医学科）；Technical University Berlin（柏林工业大学）
MedicalAgentsBench for Complex Medical Reasoning: Comparing Internalized Reasoning Models versus Externalized Agent-based Frameworks
Program in Computational Biology & Biomedical Informatics, Yale University（计算生物学与生物医学信息学项目，耶鲁大学）；Department of Biomedical Informatics & Data Science, Yale University（生物医学信息学与数据科学系，耶鲁大学）；Department of Computer Science, Yale University（计算机科学系，耶鲁大学）；Department of Biosystems Science and Engineering, ETH Zurich（生物系统科学与工程系，苏黎世联邦理工学院）；Department of Computer Science, Stanford University（计算机科学系，斯坦福大学）；Department of Bioinformatics, UT Southwestern Medical Center（生物信息学系，德克萨斯西南医学中心）
Comparing representations of long clinical texts for the task of patient note-identification
Inria（法国国家信息与自动化研究所）；Inserm（法国国家健康与医学研究院）；Data Science Platform, Imagine Institute（伊马金研究所数据科学平台）；Hôpital Universitaire Necker-Enfants Malades（内克尔-恩范茨-马拉德大学医院）；Assistance Publique - Hôpitaux de Paris (APHP)（巴黎公共医疗援助医院集团）；Université de Lorraine（洛林大学）；CNRS（法国国家科学研究中心）；LORIA（洛林计算机科学与应用实验室）；INESC-ID（里斯本大学学院计算机与系统工程研究中心）；Instituto Superior Técnico（里斯本高等技术学院）；Universidade de Lisboa（里斯本大学）
Privacy-Preserving Local Language Models for Longitudinal Data Retrieval in Chronic Dermatologic Disease: Implementation in Pemphigus Patients
Division of Systems Medicine, Department of Metabolism, Digestion and Reproduction, Imperial College London（系统医学系，代谢、消化与生殖部，帝国理工学院伦敦分校）；Department of Dermatology and Venereology, Istanbul Research and Training Hospital（皮肤科与性病科，伊斯坦布尔研究与培训医院）；Department of Dermatology and Venereology, Istanbul Medeniyet University（皮肤科与性病科，伊斯坦布尔梅德尼yet大学）；Department of Dermatology and Venereology, Istanbul Medicana Atakoy Hospital（皮肤科与性病科，伊斯坦布尔Medicana阿塔科伊医院）
Grounding Large Language Models in Clinical Evidence: A Retrieval-Augmented Generation System for Querying UK NICE Clinical Guidelines
Institute of Health Informatics, University College London（健康信息学研究所，伦敦大学学院）；Department of Biostatistics and Health Informatics, King’s College London（生物统计学与健康信息学系，伦敦国王学院）；EPSRC DRIVE-Health CDT, London, U.K.（EPSRC DRIVE-Health CDT，伦敦，英国）；Imperial College Healthcare NHS Trust（帝国学院医疗 NHS 信托）；Black Country Healthcare NHS Foundation Trust（黑斯廷斯地区医疗 NHS 基础信托）；Feldon Lane Surgery, The Dudley Group NHS Foundation Trust（费尔顿车道诊所，德比集团 NHS 基础信托）；The Cleveland Clinic, London, U.K.（克利夫兰诊所，伦敦，英国）；CogStack Limited（CogStack 有限公司）
Who Benefits From Sinus Surgery? Comparing Generative AI and Supervised Machine Learning for Predicting Surgical Outcomes in Chronic Rhinosinusitis
Department of Computer Science, Purdue University（计算机科学系，普渡大学）；Department of Computer Science, Indiana University Indianapolis（计算机科学系，印第安纳大学印第安纳波利斯分校）；Department of Otolaryngology—Head and Neck Surgery, Indiana University School of Medicine（耳鼻喉科—头颈外科，印第安纳大学医学院）
