VeriSim：一种用于在现实患者噪声下评估医疗AI的可配置框架
VeriSim: A Configurable Framework for Stress-Testing Medical AI Under Patient Communication Noise
AI总结：
本文提出VeriSim框架，通过引入可控的临床证据基础噪声，评估医疗AI在现实临床场景中的鲁棒性，发现所有模型在真实患者噪声下显著退化，且小型模型退化更严重。
摘要：
医疗大语言模型（LLMs）在标准化基准上表现优异，但这些评估未能捕捉到真实临床环境中的复杂性，如患者记忆缺失、健康素养有限、焦虑等沟通障碍。我们引入VeriSim，一种真相保持的患者模拟框架，通过混合UMLS-LLM验证机制，将可控的临床证据基础噪声注入患者响应，同时严格遵循医学地面真相。我们的框架基于同行评审的医学沟通文献，操作化了六个噪声维度，捕捉如患者记忆限制、健康素养障碍和由污名驱动的非披露等真实临床现象。在七种开放权重LLM上的实验表明，所有模型在真实患者噪声下显著退化，诊断准确率下降15-25%，对话长度增加34-55%。值得注意的是，小型模型（7B）比大型模型（70B+）退化更严重，而基于标准语料库的医疗微调对患者沟通噪声的鲁棒性提升有限。由认证临床医生评估显示，模拟质量高，具有强的标注者间一致性（kappa>0.80），而LLM-as-a-Judge作为验证的辅助评估器，实现了可扩展评估的可比可靠性。我们的结果突显了当前医疗AI中Sim-to-Real的差距。我们发布VeriSim作为开源的噪声注入框架，建立了一个严格评估临床鲁棒性的测试平台。
英文摘要：
Medical large language models are typically evaluated on idealized patient cases that do not reflect how real patients communicate. We introduce VeriSim, a patient simulation framework that injects controllable noise along six clinically grounded communication dimensions while substantially preserving each patient's medical record. Truth adherence is supported by a verifier that extracts atomic claims from each candidate utterance and judges them against a UMLS-grounded vector index built with BioLORD embeddings, using the retrieved atoms' structured clinical metadata (e.g., drug class, anatomical site, treats-condition relations) rather than surface-text similarity alone. Across seven open-weight LLMs, realistic noise reduces diagnostic accuracy by 15-25 percentage points and increases conversation length by 34-55%; the 7-8B models degrade 1.4x more than 70B+ models. A board-certified physician and a licensed nurse rate VeriSim's conversations highly on truth, realism, clinical utility, and noise fidelity (inter-annotator agreement >= 0.80 across all dimensions), and an LLM-as-judge closely tracks their ratings, enabling scalable evaluation. We release VeriSim as an open-source framework for rigorous stress-testing of clinical AI: https://github.com/mohitmarvania/VeriSim
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
