LingxiDiagBench: 用于基准测试大语言模型在中文精神科咨询与诊断中的多智能体框架
LingxiDiagBench: A Multi-Agent Framework for Benchmarking LLMs in Chinese Psychiatric Consultation and Diagnosis
- Tianqiao and Chrissy Chen Institute（天桥和克里斯西·陈研究所）
- EverMind AI Inc.（EverMind AI公司）
- Shanghai Mental Health Center, Shanghai Jiao Tong University School of Medicine（上海精神卫生中心，上海交通大学医学院）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
提出LingxiDiagBench多智能体框架，包含16K电子病历对齐的合成咨询对话数据集，评估LLM在静态诊断和动态咨询中的表现，发现其对抑郁-焦虑共病识别和12类鉴别诊断准确率低，动态咨询常不如静态评估。
AI中文摘要：
精神障碍在全球范围内高度流行，但精神科医生的短缺以及基于访谈诊断固有的主观性，对及时、一致的心理健康评估造成了重大障碍。AI辅助精神科诊断的进展受到缺乏基准测试的限制，这些基准测试需同时提供逼真的患者模拟、临床医生验证的诊断标签，并支持动态多轮咨询。我们提出LingxiDiagBench，一个大规模多智能体基准测试，评估LLM在中文静态诊断推理和动态多轮精神科咨询中的表现。其核心是LingxiDiag-16K，一个包含16,000个电子病历对齐的合成咨询对话数据集，旨在再现12个ICD-10精神科类别中真实的临床人口统计和诊断分布。通过对最先进LLM的大量实验，我们建立了关键发现：（1）尽管LLM在二元抑郁-焦虑分类上达到高准确率（高达92.3%），但在抑郁-焦虑共病识别（43.0%）和12类鉴别诊断（28.5%）上性能显著下降；（2）动态咨询通常不如静态评估，表明无效的信息收集策略显著损害下游诊断推理；（3）由LLM作为评判者评估的咨询质量与诊断准确性仅呈中等相关性，表明结构良好的提问本身并不能确保正确的诊断决策。我们发布LingxiDiag-16K和完整的评估框架，以支持可重复的研究，网址为：https://this https URL。
英文摘要：
Mental disorders are highly prevalent worldwide, but the shortage of psychiatrists and the inherent subjectivity of interview-based diagnosis create substantial barriers to timely and consistent mental-health assessment. Progress in AI-assisted psychiatric diagnosis is constrained by the absence of benchmarks that simultaneously provide realistic patient simulation, clinician-verified diagnostic labels, and support for dynamic multi-turn consultation. We present LingxiDiagBench, a large-scale multi-agent benchmark that evaluates LLMs on both static diagnostic inference and dynamic multi-turn psychiatric consultation in Chinese. At its core is LingxiDiag-16K, a dataset of 16,000 EMR-aligned synthetic consultation dialogues designed to reproduce real clinical demographic and diagnostic distributions across 12 ICD-10 psychiatric categories. Through extensive experiments across state-of-the-art LLMs, we establish key findings: (1) although LLMs achieve high accuracy on binary depression--anxiety classification (up to 92.3%), performance deteriorates substantially for depression--anxiety comorbidity recognition (43.0%) and 12-way differential diagnosis (28.5%); (2) dynamic consultation often underperforms static evaluation, indicating that ineffective information-gathering strategies significantly impair downstream diagnostic reasoning; (3) consultation quality assessed by LLM-as-a-Judge shows only moderate correlation with diagnostic accuracy, suggesting that well-structured questioning alone does not ensure correct diagnostic decisions. We release LingxiDiag-16K and the full evaluation framework to support reproducible research at https://github.com/Lingxi-mental-health/LingxiDiagBench.
