arXiv 2506.012522025-06-02cs.CLcs.AI

MTCMB：一个用于评估大语言模型在中医知识、推理与安全方面的多任务基准框架

MTCMB: A Multi-Task Benchmark Framework for Evaluating LLMs on Knowledge, Reasoning, and Safety in Traditional Chinese Medicine

School of Software Engineering, Sun Yat-sen University（中山大学软件学院）；Institute of TCM Diagnostics, Hunan University of Chinese Medicine（湖南中医药大学中医诊断研究所）；School of Artificial Intelligence, Guilin University of Electronic Technology（桂林电子科技大学人工智能学院）；Department of Computer Science, Cornell University（康奈尔大学计算机科学系）

Shufeng Kong, Xingru Yang, Yuanyuan Wei, Zijie Wang, Hao Tang, Jiuqi Qin, Shuting Lan, Yingheng Wang, Junwen Bai, Zhuangbin Chen, Zibin Zheng, Caihua Liu, Hao Liang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出MTCMB多任务基准，联合中医专家构建12个子数据集，覆盖知识、推理与安全五类任务，用于系统评估大语言模型的中医能力并揭示其在临床推理与安全合规方面的不足。

摘要：

中医（TCM）是一个拥有数千年临床经验积累的整体医学体系，在全球医疗保健中——尤其是在东亚地区——发挥着重要作用。然而，中医中隐性的推理方式、多样的文本形式以及标准化程度的不足，给计算建模和评估带来了重大挑战。大语言模型（LLMs）在包括通用医学在内的多个领域中处理自然语言方面展现出了非凡的潜力，但它们在中医领域的系统性评估仍处于发展不足的状态。现有基准要么仅狭隘地聚焦于事实性问答，要么缺乏领域特定任务和临床真实性。为了填补这一空白，我们提出了MTCMB——一个用于评估大语言模型在中医知识、推理和安全方面能力的多任务基准。MTCMB与认证中医专家合作开发，包含12个子数据集，涵盖五大类别：知识问答、语言理解、诊断推理、处方生成和安全评估。该基准整合了真实病例记录、国家执业医师考试和经典文献，为具备中医能力的模型提供了一个真实且全面的测试平台。初步结果表明，当前的大语言模型在基础知识方面表现良好，但在临床推理、处方规划和安全合规性方面存在不足。这些发现凸显了迫切需要像MTCMB这样与领域对齐的基准，以指导开发更有能力且更值得信赖的医疗人工智能系统。所有数据集、代码和评估工具均已在以下地址公开：https://github.com/Wayyuanyuan/MTCMB。

英文摘要：

Traditional Chinese Medicine (TCM) is a holistic medical system with millennia of accumulated clinical experience, playing a vital role in global healthcare-particularly across East Asia. However, the implicit reasoning, diverse textual forms, and lack of standardization in TCM pose major challenges for computational modeling and evaluation. Large Language Models (LLMs) have demonstrated remarkable potential in processing natural language across diverse domains, including general medicine. Yet, their systematic evaluation in the TCM domain remains underdeveloped. Existing benchmarks either focus narrowly on factual question answering or lack domain-specific tasks and clinical realism. To fill this gap, we introduce MTCMB-a Multi-Task Benchmark for Evaluating LLMs on TCM Knowledge, Reasoning, and Safety. Developed in collaboration with certified TCM experts, MTCMB comprises 12 sub-datasets spanning five major categories: knowledge QA, language understanding, diagnostic reasoning, prescription generation, and safety evaluation. The benchmark integrates real-world case records, national licensing exams, and classical texts, providing an authentic and comprehensive testbed for TCM-capable models. Preliminary results indicate that current LLMs perform well on foundational knowledge but fall short in clinical reasoning, prescription planning, and safety compliance. These findings highlight the urgent need for domain-aligned benchmarks like MTCMB to guide the development of more competent and trustworthy medical AI systems. All datasets, code, and evaluation tools are publicly available at: https://github.com/Wayyuanyuan/MTCMB.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。