The recently unprecedented advancements in Large Language Models (LLMs) have propelled the medical community by establishing advanced medical-domain models. However, due to the limited collection of medical datasets, there are only a few comprehensive benchmarks available to gauge progress in this area. In this paper, we introduce a new medical question-answering (QA) dataset that contains massive manual instruction for solving Traditional Chinese Medicine examination tasks, called TCMD. Specifically, our TCMD collects massive questions across diverse domains with their annotated medical subjects and thus supports us in comprehensively assessing the capability of LLMs in the TCM domain. Extensive evaluation of various general LLMs and medical-domain-specific LLMs is conducted. Moreover, we also analyze the robustness of current LLMs in solving TCM QA tasks by introducing randomness. The inconsistency of the experimental results also reveals the shortcomings of current LLMs in solving QA tasks. We also expect that our dataset can further facilitate the development of LLMs in the TCM area.
TCM-3CEval: A Triaxial Benchmark for Assessing Responses from Large Language Models in Traditional Chinese Medicine
School of Acupuncture-Moxibustion and Tuina, Shanghai University of Traditional Chinese Medicine(上海中医药大学针灸推拿学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);China Academy of Chinese Medical Sciences(中国中医科学院)
TCM-Eval: An Expert-Level Dynamic and Extensible Benchmark for Traditional Chinese Medicine
School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院);Beijing Zhimingtang Technology Co., Ltd.(北京智明堂科技有限公司);Beijing Zhiyan AI Technology Co., Ltd.(北京智言AI科技有限公司);Guangzhou University of Chinese Medicine(广州中医药大学)
DongYuan: An LLM-Based Framework for Integrative Chinese and Western Medicine Spleen-Stomach Disorders Diagnosis
Beijing Wenge Technology Co., Ltd.(北京文歌科技有限公司);Hebei Provincial Hospital of Traditional Chinese Medicine(河北省中医院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Tianjin University(天津大学)
TCM-5CEval: Extended Deep Evaluation Benchmark for LLM's Comprehensive Clinical Research Competence in Traditional Chinese Medicine
School of Artificial Intelligence in Traditional Chinese Medicine, Shanghai University of Traditional Chinese Medicine, Shanghai, China(上海中医药大学人工智能学院);Shanghai Artificial Intelligence Laboratory, Shanghai, China(上海人工智能实验室);University of Washington, Seattle, Washington, US(华盛顿大学)
Enhancing the Traditional Chinese Medicine Capabilities of Large Language Model through Reinforcement Learning from AI Feedback
School of Computer and Information Science, Southwest University(西南大学计算机与信息科学学院);School of Computing Technologies, RMIT University(RMIT大学计算技术学院);Yongdingmenwai Community Health Service Center(永定门外社区卫生服务中心)
Qibo: A Large Language Model for Traditional Chinese Medicine
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);Evidence-Based Medicine Center, Tianjin University of Traditional Chinese Medicine(天津中医药大学循证医学中心);First Teaching Hospital of Tianjin University of Traditional Chinese Medicine(天津中医药大学第一附属医院);National Clinical Research Center for Chinese Medicine Acupuncture and Moxibustion(国家中医针灸临床医学研究中心);Tiandazhitu(天大地图)
TCM-DiffRAG: Personalized Syndrome Differentiation Reasoning Method for Traditional Chinese Medicine based on Knowledge Graph and Chain of Thought
Macao Polytechnic University(澳门理工大学);Guangdong Institute of Intelligence Science and Technology(广东智能科学技术研究院);Zhejiang Chinese Medical University(浙江中医药大学);Hangzhou Ganzhicao Technology Co., Ltd(杭州甘之草科技有限公司);BoardWare Information System Limited(BoardWare信息系统有限公司)
Large Language Models Versus Physicians in Traditional Chinese Medicine: A Real-World Clinical Case Evaluation
University of Missouri(密苏里大学);Community Health Service Center Shanghai Pudong New Area(上海市浦东新区社区卫生服务中心);Jinsha County Chinese Medicine Hospital(金沙县中医院);Shanghai Seventh People's Hospital(上海市第七人民医院);The First People’s Hospital of Lanzhou City(兰州市第一人民医院);The First Affiliated Hospital of Yunnan University of Chinese Medicine(云南中医药大学第一附属医院);Northwestern University(西北大学);Linyi Traditional Chinese Medicine Hospital(临沂市中医院);Shanghai University of Traditional Chinese Medicine(上海中医药大学)
Tianyi: A Traditional Chinese Medicine all-rounder language model and its Real-World Clinical Practice
Nanjing University of Chinese Medicine(南京中医药大学);Beijing University of Chinese Medicine(北京中医药大学);Capital Medical University(首都医科大学);Zhejiang Lab(之江实验室);China Academy of Chinese Medical Sciences(中国中医科学院)
MTCMB: A Multi-Task Benchmark Framework for Evaluating LLMs on Knowledge, Reasoning, and Safety in Traditional Chinese Medicine
School of Software Engineering, Sun Yat-sen University(中山大学软件学院);Institute of TCM Diagnostics, Hunan University of Chinese Medicine(湖南中医药大学中医诊断研究所);School of Artificial Intelligence, Guilin University of Electronic Technology(桂林电子科技大学人工智能学院);Department of Computer Science, Cornell University(康奈尔大学计算机科学系)
TianHui: A Domain-Specific Large Language Model for Diverse Traditional Chinese Medicine Scenarios
School of Intelligent Medicine, Chengdu University of Traditional Chinese Medicine(成都中医药大学智能医学学院);The Acupuncture and Tuina School, Chengdu University of Traditional Chinese Medicine(成都中医药大学针灸推拿学院);Center of Preventive Medicine, Hospital of Chengdu University of Traditional Chinese Medicine(成都中医药大学附属医院预防医学中心)
Leveraging Group Relative Policy Optimization to Advance Large Language Models in Traditional Chinese Medicine
University of Missouri(密苏里大学);Christopher S. Bond Life Sciences Center, University of Missouri(密苏里大学克里斯托弗·S·邦德生命科学中心);Community Health Service Center Shanghai Pudong New Area(上海市浦东新区社区卫生服务中心);Shanghai University of Traditional Chinese Medicine(上海中医药大学)
BianCang: A Traditional Chinese Medicine Large Language Model
Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences), and with Shandong Provincial Key Laboratory of Computing Power Internet and Service Computing, Shandong Fundamental Research Center for Computer Science(计算机功率网络与信息安全重点实验室,教育部,山东计算机科学中心(国家超级计算机中心济南中心),齐鲁工业大学(山东科学院),并与山东省计算功率互联网与服务计算省级重点实验室,山东省计算机科学基础研究中心);Australian Artificial Intelligence Institute, University of Technology Sydney(澳大利亚人工智能研究所,技术悉尼大学);Affiliated Hospital of Shandong University of Traditional Chinese Medicine(山东中医药大学附属医院);College of Information Science and Technology & Artificial Intelligence, Nanjing Forestry University(信息科学与技术及人工智能学院,南京林业大学);Department of ICT and Center for AI Research, University of Agder (UiA)(ICT系和人工智能研究中心,阿格德大学(UiA))
TCMBench: A Comprehensive Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine
Shanghai Institute of AI for Education, East China Normal University(华东师范大学人工智能教育研究院);School of Pharmacy, East China Normal University(华东师范大学药学院);Shanghai Skin Disease Hospital, School of Medicine, Tongji University(同济大学医学院附属上海市皮肤病医院)
ShizhenGPT: Towards Multimodal LLMs for Traditional Chinese Medicine
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Freedom AI;National Health Data Institute, Shenzhen(深圳国家健康数据研究院);Shenzhen Research Institute of Big Data(深圳大数据研究院);Guangdong University of Technology(广东工业大学)
Jingfang: An LLM-Based Multi-Agent System for Precise Medical Consultation and Syndrome Differentiation in Traditional Chinese Medicine
Beijing University of Posts and Telecommunications(北京邮电大学);Southeast University(东南大学);University of Chinese Academy of Sciences(中国科学院大学);University of Science and Technology of China(中国科学技术大学)
Lingdan-V2: Large language models for clinically grounded reasoning in traditional Chinese medicine
School of Computer Science & Technology, Beijing Jiaotong University(北京交通大学计算机科学与技术学院);Tianjin Tasly Digital Chinese Medicine Technology Co., Ltd.(天津天士力数字中医药科技有限公司);Tasly Biopharmaceuticals Co., Ltd.(天士力生物制药有限公司);Affiliated Hospital of Hubei University of Chinese Medicine(湖北中医药大学附属医院);Da’an Health Technology (Beijing) Co., Ltd.(大安健康科技(北京)有限公司);Institute of Chinese Medicine Epidemic Disease, Beijing University of Chinese Medicine(北京中医药大学中医疫病研究所);Guang’anmen Hospital, China Academy of Chinese Medical Sciences(中国中医科学院广安门医院);Xiyuan Hospital, China Academy of Chinese Medical Sciences(中国中医科学院西苑医院)
Do "New Snow Tablets" Contain Snow? Large Language Models Over-Rely on Names to Identify Ingredients of Chinese Drugs
Liaoning University(辽宁大学);University of Queensland(昆士兰大学);National University of Singapore(新加坡国立大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);University of California, Merced(加利福尼亚大学默塞德分校)
BenCao: An Instruction-Tuned Large Language Model for Traditional Chinese Medicine
Community Health Service Center Shanghai Pudong New Area(上海浦东新区社区卫生服务中心);School of Acupuncture-Moxibustion and Tuina, Shanghai University of Traditional Chinese Medicine(上海中医药大学针灸推拿学院)
本文提出基于 ChatGPT 的中医药多模态助手 BenCao,通过自然语言指令微调、知识库、思维链模拟、专家反馈及外部 API 整合,提升诊断、药材识别与体质分类性能并实现部署。
Worlds Within Words: Translating Culture in Ancient Chinese Texts with Multi-Agent Coordination
NLP2 CT Lab, Department of Computer and Information Science, University of Macau(澳门大学自然语言处理与计算机实验室,计算机与信息科学系);Institute of International Language Services Studies, Macau Millennium College(澳门 millennium 学院国际语言服务研究学院)
PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applications
Academy for Engineering and Technology, Fudan University(复旦大学工程与技术学院);Tencent Youtu Lab(腾讯优图实验室);Cognition and Intelligent Technology Laboratory(认知与智能技术实验室);Engineering Research Center of AI and Robotics, Ministry of Education(教育部人工智能与机器人工程研究中心);AI and Unmanned Systems Engineering Research Center of Jilin Province(吉林省人工智能与无人系统工程研究中心)
Traditional Chinese Medicine Case Analysis System for High-Level Semantic Abstraction: Optimized with Prompt and RAG
Chongqing Jiudun Technology Co., Ltd.(重庆九盾科技有限公司);School of Mechanical Science & Engineering, Huazhong University of Science and Technology(华中科技大学机械科学与工程学院);Xinjiang Medical University(新疆医科大学);School of Nursing, Shanghai Jiao Tong University(上海交通大学护理学院);School of Mathematical Sciences, Shanghai Jiao Tong University(上海交通大学数学科学学院)
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
East China Normal University(华东师范大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Hasso Plattner Institute(哈索·普拉特纳研究所);University of Potsdam(波茨坦大学);Cooperative Medianet Innovation Center, Shanghai Jiao Tong University(上海交通大学合作媒体网络创新中心)
CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark
Multimodal Art Projection Research Community(多模态艺术投影研究共同体);Hong Kong University of Science and Technology(香港科技大学);University of Waterloo(滑铁卢大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Peking University(北京大学);The Hong Kong Polytechnic University(香港理工大学);Waseda University(早稻田大学);University of Manchester(曼彻斯特大学)
ZhiFangDanTai: Fine-tuning Graph-based Retrieval-Augmented Generation Model for Traditional Chinese Medicine Formula
School of Management, Capital Normal University(管理学院,首都师范大学);School of Electrical Engineering and Computer Science, the University of Queensland(电气工程与计算机科学学院,昆士兰大学)
Qilin-Med: Multi-stage Knowledge Injection Advanced Medical Large Language Model
Harvard T.H. Chan School of Public Health(哈佛陈曾熙公共卫生学院);Peking University(北京大学);University of Oxford(牛津大学);Alibaba Group(阿里巴巴集团);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Ant Group(蚂蚁集团);Tianyi Traffic Technology(天翼交通科技);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
TCM-Ladder: A Benchmark for Multimodal Question Answering on Traditional Chinese Medicine
Department of Electrical Engineering and Computer Science, University of Missouri(密苏里大学电气工程与计算机科学系);Christopher S. Bond Life Sciences Center, University of Missouri(密苏里大学克里斯托弗·S·邦德生命科学中心);Department of Computer Science, Northwestern University(西北大学计算机科学系);Department of Computer Science and Mathematics, Truman State University(特拉华州立大学计算机科学与数学系);Marquette High School(马基特高中);Community Health Service Center, Shanghai Pudong New Area(上海浦东新区社区卫生服务中心);School of Engineering and Applied Science, University of Pennsylvania(宾夕法尼亚大学工程与应用科学学院);Department of Endocrinology, Seventh People’s Hospital of Shanghai University of Traditional Chinese Medicine(上海中医药大学第七人民医院内分泌科);School of Acupuncture-Moxibustion and Tuina, Shanghai University of Traditional Chinese Medicine(上海中医药大学针灸推拿学院)
LingShu: A Large-Scale Symptom-Centric Contextualized Knowledge Graph Bridging Traditional Chinese Medicine and Modern Biomedicine
Beijing Jiaotong University(北京交通大学);Hubei Provincial Hospital of Traditional Chinese Medicine(湖北省中医院);Hubei University of Chinese Medicine(湖北中医药大学);Hubei Province Academy of Traditional Chinese Medicine(湖北省中医药研究院);China Academy of Chinese Medical Sciences(中国中医科学院);Xiyuan Hospital(西苑医院);National Clinical Research Center for Chinese Medicine Cardiology(国家中医心血管病临床医学研究中心);Hubei Provincial Clinical Research Center for Acupuncture and Moxibustion in Obesity Treatment(湖北省肥胖病针灸临床研究中心);Hubei Shizhen Laboratory(湖北时珍实验室);Tianjin Ta(天津(此处原文未完整,按原文提取))