arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.04941cs.CL
收藏

TCMD:用于评估大语言模型的中医问答数据集

TCMD: A Traditional Chinese Medicine QA Dataset for Evaluating Large Language Models

Nanjing University of Science and Technology(南京理工大学);Nanjing University of Chinese Medicine(南京中医药大学)

Ping Yu, Kaitao Song, Fengchen He, Ming Chen, Jianfeng Lu

AI总结:

本文提出中医问答数据集TCMD,通过多领域标注问题评估通用及医学专用大语言模型的中医能力,并分析其鲁棒性,以推动中医领域LLMs发展。

摘要:

大语言模型(LLMs)近期取得了前所未有的进展,通过建立先进的医学领域模型推动了医学界的发展。然而,由于医学数据集的收集有限,目前只有少数综合性基准可用于衡量该领域的进展。本文引入了一个新的医学问答(QA)数据集,其中包含大量用于解决中医考试任务的人工指令,称为TCMD。具体而言,我们的TCMD收集了跨多个领域的大量问题,并标注了其医学主题,从而支持我们全面评估LLMs在中医领域的能力。我们对各种通用LLMs和医学领域专用LLMs进行了广泛评估。此外,我们还通过引入随机性来分析当前LLMs在解决中医问答任务中的鲁棒性。实验结果的不一致性也揭示了当前LLMs在解决问答任务中的不足。我们还期望我们的数据集能够进一步促进LLMs在中医领域的发展。

英文摘要:

The recently unprecedented advancements in Large Language Models (LLMs) have propelled the medical community by establishing advanced medical-domain models. However, due to the limited collection of medical datasets, there are only a few comprehensive benchmarks available to gauge progress in this area. In this paper, we introduce a new medical question-answering (QA) dataset that contains massive manual instruction for solving Traditional Chinese Medicine examination tasks, called TCMD. Specifically, our TCMD collects massive questions across diverse domains with their annotated medical subjects and thus supports us in comprehensively assessing the capability of LLMs in the TCM domain. Extensive evaluation of various general LLMs and medical-domain-specific LLMs is conducted. Moreover, we also analyze the robustness of current LLMs in solving TCM QA tasks by introducing randomness. The inconsistency of the experimental results also reveals the shortcomings of current LLMs in solving QA tasks. We also expect that our dataset can further facilitate the development of LLMs in the TCM area.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑