面向有眼睛、耳朵和声音的对话式医疗AI
Towards Conversational Medical AI with Eyes, Ears and a Voice
AI总结：
本文提出AI co-clinician系统，利用音频视频数据实现实时临床决策，通过TelePACES评估标准显示其在管理计划和诊断差异方面接近医生，但在体格检查和疾病特异性推理上仍有不足。
摘要：
医学实践不仅依赖于熟练的对话，还依赖医生与患者之间丰富的听觉和视觉线索的细致交流与解读。基于Gemini的低延迟语音和视频处理能力，我们引入了AI co-clinician，一种首次将连续音频视频流用于实时临床决策的对话式AI系统。其双代理架构平衡了深入的临床推理与自然对话所需的低延迟。我们通过视频接口模拟远程医疗咨询进行评估，设计了20个标准化门诊场景和
英文摘要：
The practice of medicine relies not only upon skillful dialogue but also on the nuanced exchange and interpretation of rich auditory and visual cues between doctors and patients. Building on the low-latency voice and video processing capabilities of Gemini, we introduce AI co-clinician, a first-of-its-kind conversational AI system utilizing continuous streams of audio-visual data from live patient conversations to inform real-time clinical decisions. Its dual-agent architecture balances deep clinical reasoning with the low latency required for natural dialogue. To assess this system, we implemented a video-based interface emulating telemedicine consultations. We crafted 20 standardized outpatient scenarios requiring proactive real-time auditory and visual reasoning and designed "TelePACES" evaluation criteria alongside case-specific rubrics. In a randomized, interface-blinded, crossover simulation study (n = 120 encounters) with 10 internal medicine residents as patient actors, we compared AI co-clinician with primary care physicians (PCPs), GPT-Realtime, and a baseline agent. AI co-clinician approached PCPs in key TelePACES dimensions, including management plans and differential diagnosis, while significantly outperforming GPT-Realtime across all general criteria. While our agent demonstrated parity with PCPs in case-specific triage measures, physicians maintained superior overall performance in case-specific assessments. Although AI co-clinician marks a significant advance in real-time telemedical AI, gaps remain in physical examination and disease-specific reasoning. Our work shows that text-only approaches fail to capture the true challenges of medical consultation and suggests that high-stakes real-time diagnostic AI is most safely advanced in collaborative, triadic models where AI can be a supportive co-clinician for doctors and patients.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
