通知：联邦节假日调休：哥伦布日 / 原住民日（Columbus Day / Indigenous Peoples' Day），今日（13日）无更新。通知：联邦节假日调休：哥伦布日 / 原住民日（Columbus Day / Indigenous Peoples' Day），今日（13日）无更新。
arXiv 2505.04653cs.CLcs.AIcs.CVcs.LG
利用多模态推理推进对话式诊断人工智能
Advancing Conversational Diagnostic AI with Multimodal Reasoning
Google（谷歌）；DeepMind（深Mind）；Google Research（谷歌研究院）
Khaled Saab, Jan Freyberg, Chunjong Park, Tim Strother, Yong Cheng, Wei-Hung Weng, David G. T. Barrett, David Stutz, Nenad Tomasev, Anil Palepu, Valentin Liévin… 展开作者
Khaled Saab, Jan Freyberg, Chunjong Park, Tim Strother, Yong Cheng, Wei-Hung Weng, David G. T. Barrett, David Stutz, Nenad Tomasev, Anil Palepu, Valentin Liévin, Yash Sharma, Roma Ruparel, Abdullah Ahmed, Elahe Vedadi, Kimberly Kanada, Cian Hughes, Yun Liu, Geoff Brown, Yang Gao, Sean Li, S. Sara Mahdavi, James Manyika, Katherine Chou, Yossi Matias, Avinatan Hassidim, Dale R. Webster, Pushmeet Kohli, S. M. Ali Eslami, Joëlle Barral, Adam Rodman, Vivek Natarajan, Mike Schaekermann, Tao Tu, Alan Karthikesalingam, Ryutaro Tanno
Large Language Models (LLMs) have demonstrated great potential for conducting diagnostic conversations but evaluation has been largely limited to language-only interactions, deviating from the real-world requirements of remote care delivery. Instant messaging platforms permit clinicians and patients to upload and discuss multimodal medical artifacts seamlessly in medical consultation, but the ability of LLMs to reason over such data while preserving other attributes of competent diagnostic conversation remains unknown. Here we advance the conversational diagnosis and management performance of the Articulate Medical Intelligence Explorer (AMIE) through a new capability to gather and interpret multimodal data, and reason about this precisely during consultations. Leveraging Gemini 2.0 Flash, our system implements a state-aware dialogue framework, where conversation flow is dynamically controlled by intermediate model outputs reflecting patient states and evolving diagnoses. Follow-up questions are strategically directed by uncertainty in such patient states, leading to a more structured multimodal history-taking process that emulates experienced clinicians. We compared AMIE to primary care physicians (PCPs) in a randomized, blinded, OSCE-style study of chat-based consultations with patient actors. We constructed 105 evaluation scenarios using artifacts like smartphone skin photos, ECGs, and PDFs of clinical documents across diverse conditions and demographics. Our rubric assessed multimodal capabilities and other clinically meaningful axes like history-taking, diagnostic accuracy, management reasoning, communication, and empathy. Specialist evaluation showed AMIE to be superior to PCPs on 7/9 multimodal and 29/32 non-multimodal axes (including diagnostic accuracy). The results show clear progress in multimodal conversational diagnostic AI, but real-world translation needs further research.
RadVLM: A Multitask Conversational Vision-Language Model for Radiology
Department of Radiology, Kobe University（金泽大学放射科）；Department of Computer Science, ETH Zurich（苏黎世联邦理工学院计算机科学系）；Department of Advanced Imaging in Medical Magnetic Resonance, Kyoto University（京都大学医学磁共振高级成像部门）；Department of Quantitative Biomedicine, University of Zurich（苏黎世大学定量生物医学系）；Diagnostic and Interventional Radiology, University Hospital Zurich（苏黎世大学医院诊断与介入放射科）
LingLan: An Advancing Traditional Chinese Medicine Diagnosis LLM with Multimodal Data
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Guangdong Provincial Laboratory of Traditional Chinese Medicine Hengqin（广东省中医药横琴实验室）
Towards Explainable Conversational AI for Early Diagnosis with Large Language Models
Department of Information and Communication Technology（信息与通信技术系）；Bangladesh University of Professionals（孟加拉专业大学）；Institute of Information Technology（信息技术研究所）；Jahangirnagar University（贾汗吉尔纳瓦德大学）
RetinalGPT: A Retinal Clinical Preference Conversational Assistant Powered by Large Vision-Language Models
School of Computing and Augmented Intelligence, Arizona State University（亚利桑那州立大学计算与增强智能学院）；School of Computing, Clemson University（克莱姆森大学计算机学院）；Washington University School of Medicine in St. Louis（圣路易斯华盛顿大学医学院）；The Keck School of Medicine of the University of Southern California（南加州大学凯克医学院）；Mayo Clinic（梅奥诊所）；Banner Alzheimer’s Institute（班纳阿尔茨海默病研究所）
Systematic Exploration of Dialogue Summarization Approaches for Reproducibility, Comparative Assessment, and Methodological Innovations for Advancing Natural Language Processing in Abstractive Summarization
SRM Institute of Science and Technology（SRM科学技术学院）；SRM University, AP（SRM大学（安得拉邦））
Eco-Bee: A Personalised Multi-Modal Agent for Advancing Student Climate Awareness and Sustainable Behaviour in Campus Ecosystems
University of Hull（赫尔大学）；The Spaceship Academy（太空学院）；City St George’s, University of London（伦敦大学城市学院）；King’s College London（伦敦国王学院）；Lancaster University（兰卡斯特大学）；University of Oxford（牛津大学）
Building open domain conversational systems that allow users to have engaging conversations on topics of their choice is a challenging task. Alexa Prize was launched in 2016 to tackle the problem of achieving natural, sustained, coherent and engaging open-domain dialogs. In the second iteration of the competition in 2018, university teams advanced the state of the art by using context in dialog models, leveraging knowledge graphs for language understanding, handling complex utterances, building statistical and hierarchical dialog managers, and leveraging model-driven signals from user response…
This paper presents a novel streaming automatic speech recognition (ASR) framework for multi-talker overlapping speech captured by a distant microphone array with an arbitrary geometry. Our framework, named t-SOT-VA, capitalizes on independently developed two recent technologies; array-geometry-agnostic continuous speech separation, or VarArray, and streaming multi-talker ASR based on token-level serialized output training (t-SOT). To combine the best of both technologies, we newly design a t-SOT-based ASR model that generates a serialized multi-talker transcription based on two separated spee…
Interpretable Droplet Digital PCR Assay for Trustworthy Molecular Diagnostics
The Chinese University of Hong Kong（香港中文大学）；University of California, Los Angeles（加州大学洛杉矶分校）；Guangdong Institute of Intelligence Science and Technology（广东省智能科学与技术研究院）；Hainan University（海南大学）；The University of Hong Kong（香港大学）；Tsinghua University（清华大学）
