arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.01779cs.AI
收藏

LingLanMiDian: LLMs在中医知识与临床推理上的系统评估

LingLanMedBench: A Unified Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine

Department of Computer Science and Technology, Beijing Jiaotong University, Beijing, China(北京交通大学计算机科学与技术学院);Tianjin Tasly Digital Chinese Medicine Technology Co., Ltd.(天津塔斯丽数字中医科技有限公司);Tasly Biopharmaceuticals Co., Ltd.(塔斯丽生物医药有限公司);State Key Laboratory of Chinese Medicine Modernization, Tianjin, China(中药现代化国家工程实验室,天津,中国);Institute of Liver Diseases, Hubei Key Laboratory of the theory and application research of liver and kidney in traditional Chinese medicine, Hubei Provincial Hospital of Traditional Chinese Medicine, Wuhan, China(肝病研究所,湖北省中医肝肾理论与应用研究重点实验室,湖北省中医药研究院,武汉,中国);Affiliated Hospital of Hubei University of Chinese Medicine, Wuhan, China(湖北中医药大学附属医院,武汉,中国);Hubei Province Academy of Traditional Chinese Medicine, Wuhan, China(湖北省中医药研究院,武汉,中国);Department of Gastroenterology, Guang’anmen Hospital, China Academy of Chinese Medical Sciences, Beijing, China(消化内科,广安门医院,中国中医科学院,北京,中国);China Academy of Chinese Medical Sciences, Beijing, China(中国中医科学院,北京,中国);China Institute for History of Medicine and Medical Literature, China Academy of Chinese Medical Sciences, Beijing, China(中国中医科学院中国医学史与医学文献研究所,北京,中国);Beijing Research Institute of Chinese Medicine, Beijing University of Chinese Medicine, Beijing, China(北京中医研究院,北京中医药大学,北京,中国);Beijing University of Chinese Medicine Third Affiliated Hospital, Beijing University of Chinese Medicine, Beijing 100029, China(北京中医药大学第三附属医院,北京中医药大学,北京100029,中国)
Rui Hua, Yu Wei, Zixin Shu, Kai Chang, Dengying Yan, Jianan Xia, Hui Zhu, Xiaodong Li, Dongmei Jia, Zhuye Gao, Yanyan Meng, Haibin Yu, Fei Dong, Kan Ding, Runsh… 展开作者

Rui Hua, Yu Wei, Zixin Shu, Kai Chang, Dengying Yan, Jianan Xia, Hui Zhu, Xiaodong Li, Dongmei Jia, Zhuye Gao, Yanyan Meng, Haibin Yu, Fei Dong, Kan Ding, Runshun Zhang, Jian Yu, Jiansheng Li, Baoyan Liu, Wenjia Wang, Xuezhong Zhou

AI总结:

LingLan基准测试系统评估了LLMs在中医知识和临床推理上的表现,揭示了当前模型与专家在专门推理上的差距。

摘要:

大型语言模型(LLMs)在医学NLP领域发展迅速,但具有独特本体、术语和推理模式的中医(TCM)需要领域忠实的评估。现有的中医基准测试在覆盖范围和规模上碎片化,并且依赖于非统一或生成密集型的评分,这阻碍了公平的比较。我们提出了LingLanMiDian(LingLan)基准测试,这是一个大规模、专家编纂、多任务套件,它在知识回忆、多跳推理、信息提取和现实世界临床决策中统一评估。LingLan引入了一致的度量设计,一种容忍同义词的临床标签协议,每个数据集400项的困难子集,以及将诊断和治疗推荐重新解释为单选决策识别。我们对14种领先的开源和专有LLMs进行了全面的零样本评估,提供了对它们在中医常识知识理解、推理和临床决策支持中的优势和局限性的统一视角;关键的是,对困难子集的评估揭示了当前模型与中医专家在专门推理方面的显著差距。通过标准化评估将基本知识与应用推理联系起来,LingLan建立了统一、定量且可扩展的基础,以推进中医LLMs和领域特定的医学AI研究。所有评估数据和代码均可在https://github.com/TCMAI-BJTU/LingLan和http://tcmnlp.com上获得。

英文摘要:

Reliable evaluation of large language models in traditional Chinese medicine remains limited because existing benchmarks mainly focus on examination-style question answering or isolated generative tasks. We developed LingLanMedBench, a unified benchmark comprising 25,620 expert-verified items across 13 subtasks and five domains, covering knowledge, language understanding, information extraction, and diagnostic-therapeutic decision-making. LingLanMedBench standardizes task formats and evaluation metrics and further incorporates synonym-tolerant clinical-label matching, empirically derived hard subsets, and decision-recognition reformulations of open-ended clinical tasks. Fourteen open-source and proprietary models were evaluated under a standardized zero-shot protocol. Performance varied substantially across domains, model families, task formulations, and difficulty levels. The leading overall score decreased from 51.1 on the full benchmark to 31.9 on the hard subsets. Prescription performance also differed markedly between open-ended generation, with F1 scores of 32.7 and 20.3, and decision recognition, with accuracies of 90.6 and 56.8 on the full and hard subsets, respectively. These findings reveal a substantial gap between knowledge-oriented performance and clinically faithful reasoning. LingLanMedBench provides a standardized and reproducible framework for characterizing current capabilities, identifying systematic limitations, and tracking meaningful progress in large language models for traditional Chinese medicine.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑