arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.01126cs.CLcs.AI
收藏

TCMBench:一个用于评估大型语言模型在中医学领域的综合基准

TCMBench: A Comprehensive Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine

Shanghai Institute of AI for Education, East China Normal University(华东师范大学人工智能教育研究院);School of Pharmacy, East China Normal University(华东师范大学药学院);Shanghai Skin Disease Hospital, School of Medicine, Tongji University(同济大学医学院附属上海市皮肤病医院)

Wenjing Yue, Xiaoling Wang, Wei Zhu, Ming Guan, Huanran Zheng, Pengfei Wang, Changzhi Sun, Xin Ma

AI总结:

本文提出TCM-Bench基准及TCM-ED数据集评估大模型在中医领域的表现,并设计TCMScore指标衡量答案质量,实验揭示了现有模型的局限及领域知识的影响。

摘要:

大型语言模型(LLMs)通过基准测试在包括西医领域在内的各种自然语言处理任务中表现优异。然而,针对中医学(TCM)这一历史悠久且影响深远的领域,LLMs的专业评估基准尚未被覆盖。为了填补这一研究空白,我们引入了TCM-Bench,一个用于评估LLMs在TCM领域表现的综合基准。它包含TCM-ED数据集,由源自中医执业医师考试(TCMLE)的5,473道题目组成,其中包括1,300道带有权威解析的题目。它涵盖了TCMLE的核心组成部分,包括中医基础和临床实践。为了在问答准确性之外评估LLMs,我们提出了TCMScore,这是一个专门为评估LLMs针对TCM相关问题生成答案质量而量身定制的指标。它综合考虑了TCM语义和知识的一致性。在从多个视角进行综合实验分析后,我们得出以下发现:(1)LLMs在该基准上的表现不佳,凸显了其在TCM领域仍有巨大的改进空间。(2)引入领域知识可以提升LLMs的性能。然而,对于像ZhongJing-TCM这样的领域内模型,其生成的解析文本质量有所下降,我们假设其微调过程影响了基础LLM能力。(3)诸如Rouge和BertScore等传统的文本生成质量指标容易受文本长度和表面语义歧义的影响,而像TCMScore这样的领域特定指标则能进一步补充和解释它们的评估结果。这些发现突显了LLMs在TCM中的能力与局限,旨在为医学研究提供更深刻的帮助。

英文摘要:

Large language models (LLMs) have performed remarkably well in various natural language processing tasks by benchmarking, including in the Western medical domain. However, the professional evaluation benchmarks for LLMs have yet to be covered in the traditional Chinese medicine(TCM) domain, which has a profound history and vast influence. To address this research gap, we introduce TCM-Bench, an comprehensive benchmark for evaluating LLM performance in TCM. It comprises the TCM-ED dataset, consisting of 5,473 questions sourced from the TCM Licensing Exam (TCMLE), including 1,300 questions with authoritative analysis. It covers the core components of TCMLE, including TCM basis and clinical practice. To evaluate LLMs beyond accuracy of question answering, we propose TCMScore, a metric tailored for evaluating the quality of answers generated by LLMs for TCM related questions. It comprehensively considers the consistency of TCM semantics and knowledge. After conducting comprehensive experimental analyses from diverse perspectives, we can obtain the following findings: (1) The unsatisfactory performance of LLMs on this benchmark underscores their significant room for improvement in TCM. (2) Introducing domain knowledge can enhance LLMs' performance. However, for in-domain models like ZhongJing-TCM, the quality of generated analysis text has decreased, and we hypothesize that their fine-tuning process affects the basic LLM capabilities. (3) Traditional metrics for text generation quality like Rouge and BertScore are susceptible to text length and surface semantic ambiguity, while domain-specific metrics such as TCMScore can further supplement and explain their evaluation results. These findings highlight the capabilities and limitations of LLMs in the TCM and aim to provide a more profound assistance to medical research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑