TCMBench: A Comprehensive Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine
Shanghai Institute of AI for Education, East China Normal University(华东师范大学人工智能教育研究院);School of Pharmacy, East China Normal University(华东师范大学药学院);Shanghai Skin Disease Hospital, School of Medicine, Tongji University(同济大学医学院附属上海市皮肤病医院)
Wenjing Yue, Xiaoling Wang, Wei Zhu, Ming Guan, Huanran Zheng, Pengfei Wang, Changzhi Sun, Xin Ma
Large language models (LLMs) have performed remarkably well in various natural language processing tasks by benchmarking, including in the Western medical domain. However, the professional evaluation benchmarks for LLMs have yet to be covered in the traditional Chinese medicine(TCM) domain, which has a profound history and vast influence. To address this research gap, we introduce TCM-Bench, an comprehensive benchmark for evaluating LLM performance in TCM. It comprises the TCM-ED dataset, consisting of 5,473 questions sourced from the TCM Licensing Exam (TCMLE), including 1,300 questions with authoritative analysis. It covers the core components of TCMLE, including TCM basis and clinical practice. To evaluate LLMs beyond accuracy of question answering, we propose TCMScore, a metric tailored for evaluating the quality of answers generated by LLMs for TCM related questions. It comprehensively considers the consistency of TCM semantics and knowledge. After conducting comprehensive experimental analyses from diverse perspectives, we can obtain the following findings: (1) The unsatisfactory performance of LLMs on this benchmark underscores their significant room for improvement in TCM. (2) Introducing domain knowledge can enhance LLMs' performance. However, for in-domain models like ZhongJing-TCM, the quality of generated analysis text has decreased, and we hypothesize that their fine-tuning process affects the basic LLM capabilities. (3) Traditional metrics for text generation quality like Rouge and BertScore are susceptible to text length and surface semantic ambiguity, while domain-specific metrics such as TCMScore can further supplement and explain their evaluation results. These findings highlight the capabilities and limitations of LLMs in the TCM and aim to provide a more profound assistance to medical research.
Enhancing the Traditional Chinese Medicine Capabilities of Large Language Model through Reinforcement Learning from AI Feedback
School of Computer and Information Science, Southwest University(西南大学计算机与信息科学学院);School of Computing Technologies, RMIT University(RMIT大学计算技术学院);Yongdingmenwai Community Health Service Center(永定门外社区卫生服务中心)
Qibo: A Large Language Model for Traditional Chinese Medicine
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);Evidence-Based Medicine Center, Tianjin University of Traditional Chinese Medicine(天津中医药大学循证医学中心);First Teaching Hospital of Tianjin University of Traditional Chinese Medicine(天津中医药大学第一附属医院);National Clinical Research Center for Chinese Medicine Acupuncture and Moxibustion(国家中医针灸临床医学研究中心);Tiandazhitu(天大地图)
TCM-DiffRAG: Personalized Syndrome Differentiation Reasoning Method for Traditional Chinese Medicine based on Knowledge Graph and Chain of Thought
Macao Polytechnic University(澳门理工大学);Guangdong Institute of Intelligence Science and Technology(广东智能科学技术研究院);Zhejiang Chinese Medical University(浙江中医药大学);Hangzhou Ganzhicao Technology Co., Ltd(杭州甘之草科技有限公司);BoardWare Information System Limited(BoardWare信息系统有限公司)
ZhiFangDanTai: Fine-tuning Graph-based Retrieval-Augmented Generation Model for Traditional Chinese Medicine Formula
School of Management, Capital Normal University(管理学院,首都师范大学);School of Electrical Engineering and Computer Science, the University of Queensland(电气工程与计算机科学学院,昆士兰大学)
TCM-3CEval: A Triaxial Benchmark for Assessing Responses from Large Language Models in Traditional Chinese Medicine
School of Acupuncture-Moxibustion and Tuina, Shanghai University of Traditional Chinese Medicine(上海中医药大学针灸推拿学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);China Academy of Chinese Medical Sciences(中国中医科学院)
Tianyi: A Traditional Chinese Medicine all-rounder language model and its Real-World Clinical Practice
Nanjing University of Chinese Medicine(南京中医药大学);Beijing University of Chinese Medicine(北京中医药大学);Capital Medical University(首都医科大学);Zhejiang Lab(之江实验室);China Academy of Chinese Medical Sciences(中国中医科学院)
MTCMB: A Multi-Task Benchmark Framework for Evaluating LLMs on Knowledge, Reasoning, and Safety in Traditional Chinese Medicine
School of Software Engineering, Sun Yat-sen University(中山大学软件学院);Institute of TCM Diagnostics, Hunan University of Chinese Medicine(湖南中医药大学中医诊断研究所);School of Artificial Intelligence, Guilin University of Electronic Technology(桂林电子科技大学人工智能学院);Department of Computer Science, Cornell University(康奈尔大学计算机科学系)
LingLan: An Advancing Traditional Chinese Medicine Diagnosis LLM with Multimodal Data
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Guangdong Provincial Laboratory of Traditional Chinese Medicine Hengqin(广东省中医药横琴实验室)
TianHui: A Domain-Specific Large Language Model for Diverse Traditional Chinese Medicine Scenarios
School of Intelligent Medicine, Chengdu University of Traditional Chinese Medicine(成都中医药大学智能医学学院);The Acupuncture and Tuina School, Chengdu University of Traditional Chinese Medicine(成都中医药大学针灸推拿学院);Center of Preventive Medicine, Hospital of Chengdu University of Traditional Chinese Medicine(成都中医药大学附属医院预防医学中心)
TCM-Eval: An Expert-Level Dynamic and Extensible Benchmark for Traditional Chinese Medicine
School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院);Beijing Zhimingtang Technology Co., Ltd.(北京智明堂科技有限公司);Beijing Zhiyan AI Technology Co., Ltd.(北京智言AI科技有限公司);Guangzhou University of Chinese Medicine(广州中医药大学)
BianCang: A Traditional Chinese Medicine Large Language Model
Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences), and with Shandong Provincial Key Laboratory of Computing Power Internet and Service Computing, Shandong Fundamental Research Center for Computer Science(计算机功率网络与信息安全重点实验室,教育部,山东计算机科学中心(国家超级计算机中心济南中心),齐鲁工业大学(山东科学院),并与山东省计算功率互联网与服务计算省级重点实验室,山东省计算机科学基础研究中心);Australian Artificial Intelligence Institute, University of Technology Sydney(澳大利亚人工智能研究所,技术悉尼大学);Affiliated Hospital of Shandong University of Traditional Chinese Medicine(山东中医药大学附属医院);College of Information Science and Technology & Artificial Intelligence, Nanjing Forestry University(信息科学与技术及人工智能学院,南京林业大学);Department of ICT and Center for AI Research, University of Agder (UiA)(ICT系和人工智能研究中心,阿格德大学(UiA))
DongYuan: An LLM-Based Framework for Integrative Chinese and Western Medicine Spleen-Stomach Disorders Diagnosis
Beijing Wenge Technology Co., Ltd.(北京文歌科技有限公司);Hebei Provincial Hospital of Traditional Chinese Medicine(河北省中医院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Tianjin University(天津大学)
Leveraging Group Relative Policy Optimization to Advance Large Language Models in Traditional Chinese Medicine
University of Missouri(密苏里大学);Christopher S. Bond Life Sciences Center, University of Missouri(密苏里大学克里斯托弗·S·邦德生命科学中心);Community Health Service Center Shanghai Pudong New Area(上海市浦东新区社区卫生服务中心);Shanghai University of Traditional Chinese Medicine(上海中医药大学)
Lingdan-V2: Large language models for clinically grounded reasoning in traditional Chinese medicine
School of Computer Science & Technology, Beijing Jiaotong University(北京交通大学计算机科学与技术学院);Tianjin Tasly Digital Chinese Medicine Technology Co., Ltd.(天津天士力数字中医药科技有限公司);Tasly Biopharmaceuticals Co., Ltd.(天士力生物制药有限公司);Affiliated Hospital of Hubei University of Chinese Medicine(湖北中医药大学附属医院);Da’an Health Technology (Beijing) Co., Ltd.(大安健康科技(北京)有限公司);Institute of Chinese Medicine Epidemic Disease, Beijing University of Chinese Medicine(北京中医药大学中医疫病研究所);Guang’anmen Hospital, China Academy of Chinese Medical Sciences(中国中医科学院广安门医院);Xiyuan Hospital, China Academy of Chinese Medical Sciences(中国中医科学院西苑医院)
Large Language Models Versus Physicians in Traditional Chinese Medicine: A Real-World Clinical Case Evaluation
University of Missouri(密苏里大学);Community Health Service Center Shanghai Pudong New Area(上海市浦东新区社区卫生服务中心);Jinsha County Chinese Medicine Hospital(金沙县中医院);Shanghai Seventh People's Hospital(上海市第七人民医院);The First People’s Hospital of Lanzhou City(兰州市第一人民医院);The First Affiliated Hospital of Yunnan University of Chinese Medicine(云南中医药大学第一附属医院);Northwestern University(西北大学);Linyi Traditional Chinese Medicine Hospital(临沂市中医院);Shanghai University of Traditional Chinese Medicine(上海中医药大学)
Jingfang: An LLM-Based Multi-Agent System for Precise Medical Consultation and Syndrome Differentiation in Traditional Chinese Medicine
Beijing University of Posts and Telecommunications(北京邮电大学);Southeast University(东南大学);University of Chinese Academy of Sciences(中国科学院大学);University of Science and Technology of China(中国科学技术大学)
ShizhenGPT: Towards Multimodal LLMs for Traditional Chinese Medicine
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Freedom AI;National Health Data Institute, Shenzhen(深圳国家健康数据研究院);Shenzhen Research Institute of Big Data(深圳大数据研究院);Guangdong University of Technology(广东工业大学)
TCM-5CEval: Extended Deep Evaluation Benchmark for LLM's Comprehensive Clinical Research Competence in Traditional Chinese Medicine
School of Artificial Intelligence in Traditional Chinese Medicine, Shanghai University of Traditional Chinese Medicine, Shanghai, China(上海中医药大学人工智能学院);Shanghai Artificial Intelligence Laboratory, Shanghai, China(上海人工智能实验室);University of Washington, Seattle, Washington, US(华盛顿大学)
LingLanMedBench: A Unified Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine
Department of Computer Science and Technology, Beijing Jiaotong University, Beijing, China(北京交通大学计算机科学与技术学院);Tianjin Tasly Digital Chinese Medicine Technology Co., Ltd.(天津塔斯丽数字中医科技有限公司);Tasly Biopharmaceuticals Co., Ltd.(塔斯丽生物医药有限公司);State Key Laboratory of Chinese Medicine Modernization, Tianjin, China(中药现代化国家工程实验室,天津,中国);Institute of Liver Diseases, Hubei Key Laboratory of the theory and application research of liver and kidney in traditional Chinese medicine, Hubei Provincial Hospital of Traditional Chinese Medicine, Wuhan, China(肝病研究所,湖北省中医肝肾理论与应用研究重点实验室,湖北省中医药研究院,武汉,中国);Affiliated Hospital of Hubei University of Chinese Medicine, Wuhan, China(湖北中医药大学附属医院,武汉,中国);Hubei Province Academy of Traditional Chinese Medicine, Wuhan, China(湖北省中医药研究院,武汉,中国);Department of Gastroenterology, Guang’anmen Hospital, China Academy of Chinese Medical Sciences, Beijing, China(消化内科,广安门医院,中国中医科学院,北京,中国);China Academy of Chinese Medical Sciences, Beijing, China(中国中医科学院,北京,中国);China Institute for History of Medicine and Medical Literature, China Academy of Chinese Medical Sciences, Beijing, China(中国中医科学院中国医学史与医学文献研究所,北京,中国);Beijing Research Institute of Chinese Medicine, Beijing University of Chinese Medicine, Beijing, China(北京中医研究院,北京中医药大学,北京,中国);Beijing University of Chinese Medicine Third Affiliated Hospital, Beijing University of Chinese Medicine, Beijing 100029, China(北京中医药大学第三附属医院,北京中医药大学,北京100029,中国)
BenCao: An Instruction-Tuned Large Language Model for Traditional Chinese Medicine
Community Health Service Center Shanghai Pudong New Area(上海浦东新区社区卫生服务中心);School of Acupuncture-Moxibustion and Tuina, Shanghai University of Traditional Chinese Medicine(上海中医药大学针灸推拿学院)
本文提出基于 ChatGPT 的中医药多模态助手 BenCao,通过自然语言指令微调、知识库、思维链模拟、专家反馈及外部 API 整合,提升诊断、药材识别与体质分类性能并实现部署。
Evidence-Based Intelligent Diagnostic and Therapeutic Visualization System with Large Language Models: Multi-Turn Interaction and Multimodal Treatment Plan Generation
Harbin Institute of Technology, Weihai(哈尔滨工业大学(威海));Harbin Institute of Technology (Weihai) Qingdao Research Institute(哈尔滨工业大学(威海)青岛研究院);Shandong Key Laboratory of Digital Service Computing Technology and Systems(山东省数字服务计算技术与系统重点实验室);Weihai Municipal Hospital(威海市人民医院);Shanghai Taizhu Technology Co., Ltd(上海泰山技术有限公司);Tianjin Zhifu Qihuang Medical Technology Co., Ltd(天津中孚启黄医疗技术有限公司)
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
East China Normal University(华东师范大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Hasso Plattner Institute(哈索·普拉特纳研究所);University of Potsdam(波茨坦大学);Cooperative Medianet Innovation Center, Shanghai Jiao Tong University(上海交通大学合作媒体网络创新中心)
Do "New Snow Tablets" Contain Snow? Large Language Models Over-Rely on Names to Identify Ingredients of Chinese Drugs
Liaoning University(辽宁大学);University of Queensland(昆士兰大学);National University of Singapore(新加坡国立大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);University of California, Merced(加利福尼亚大学默塞德分校)
CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark
Multimodal Art Projection Research Community(多模态艺术投影研究共同体);Hong Kong University of Science and Technology(香港科技大学);University of Waterloo(滑铁卢大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Peking University(北京大学);The Hong Kong Polytechnic University(香港理工大学);Waseda University(早稻田大学);University of Manchester(曼彻斯特大学)
Worlds Within Words: Translating Culture in Ancient Chinese Texts with Multi-Agent Coordination
NLP2 CT Lab, Department of Computer and Information Science, University of Macau(澳门大学自然语言处理与计算机实验室,计算机与信息科学系);Institute of International Language Services Studies, Macau Millennium College(澳门 millennium 学院国际语言服务研究学院)
Traditional Chinese Medicine Case Analysis System for High-Level Semantic Abstraction: Optimized with Prompt and RAG
Chongqing Jiudun Technology Co., Ltd.(重庆九盾科技有限公司);School of Mechanical Science & Engineering, Huazhong University of Science and Technology(华中科技大学机械科学与工程学院);Xinjiang Medical University(新疆医科大学);School of Nursing, Shanghai Jiao Tong University(上海交通大学护理学院);School of Mathematical Sciences, Shanghai Jiao Tong University(上海交通大学数学科学学院)
PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applications
Academy for Engineering and Technology, Fudan University(复旦大学工程与技术学院);Tencent Youtu Lab(腾讯优图实验室);Cognition and Intelligent Technology Laboratory(认知与智能技术实验室);Engineering Research Center of AI and Robotics, Ministry of Education(教育部人工智能与机器人工程研究中心);AI and Unmanned Systems Engineering Research Center of Jilin Province(吉林省人工智能与无人系统工程研究中心)
LingShu: A Large-Scale Symptom-Centric Contextualized Knowledge Graph Bridging Traditional Chinese Medicine and Modern Biomedicine
Beijing Jiaotong University(北京交通大学);Hubei Provincial Hospital of Traditional Chinese Medicine(湖北省中医院);Hubei University of Chinese Medicine(湖北中医药大学);Hubei Province Academy of Traditional Chinese Medicine(湖北省中医药研究院);China Academy of Chinese Medical Sciences(中国中医科学院);Xiyuan Hospital(西苑医院);National Clinical Research Center for Chinese Medicine Cardiology(国家中医心血管病临床医学研究中心);Hubei Provincial Clinical Research Center for Acupuncture and Moxibustion in Obesity Treatment(湖北省肥胖病针灸临床研究中心);Hubei Shizhen Laboratory(湖北时珍实验室);Tianjin Ta(天津(此处原文未完整,按原文提取))