arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.12945cs.CL
收藏

ExplainCPE:中文药师考试自由文本解释基准

ExplainCPE: A Free-text Explanation Benchmark of Chinese Pharmacist Examination

Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))

Dongfang Li, Jindi Yu, Baotian Hu, Zhenran Xu, Min Zhang

AI总结:

本文提出中文医学解释基准ExplainCPE(超7k实例),通过分析ChatGPT和GPT-4的错误揭示LLMs在文本理解与计算推理上的局限,并发现不同模型对上下文学习偏好各异,为评估LLMs医学可解释性迈出第一步。

摘要:

随着ChatGPT和GPT-4引领大语言模型(LLMs)的发展,越来越多的研究者正在考察它们在各类任务上的表现。但在LLMs的可解释性能力方面,即在给出答案后生成理由的能力,仍需更多研究。现有的解释数据集大多是英文通用知识问答,导致主题和语言多样性不足。为解决生成理由问答数据集中的语言偏见和医学资源匮乏问题,我们提出了ExplainCPE(超过7k个实例),这是一个具有挑战性的简体中文医学基准。我们分析了ChatGPT和GPT-4的错误,指出当前LLMs在文本理解和计算推理方面的局限性。在实验过程中,我们还发现不同的LLMs对上下文学习有不同的偏好。ExplainCPE提出了重大挑战,但其进一步研究的潜力值得期待,并且它可以用于评估模型生成解释的能力。AI安全性和可信度需要更多关注,本工作迈出了探索LLMs医学可解释性的第一步。数据集可在https://github.com/HITsz-TMG/ExplainCPE获取。

英文摘要:

As ChatGPT and GPT-4 spearhead the development of Large Language Models (LLMs), more researchers are investigating their performance across various tasks. But more research needs to be done on the interpretability capabilities of LLMs, that is, the ability to generate reasons after an answer has been given. Existing explanation datasets are mostly English-language general knowledge questions, which leads to insufficient thematic and linguistic diversity. To address the language bias and lack of medical resources in generating rationales QA datasets, we present ExplainCPE (over 7k instances), a challenging medical benchmark in Simplified Chinese. We analyzed the errors of ChatGPT and GPT-4, pointing out the limitations of current LLMs in understanding text and computational reasoning. During the experiment, we also found that different LLMs have different preferences for in-context learning. ExplainCPE presents a significant challenge, but its potential for further investigation is promising, and it can be used to evaluate the ability of a model to generate explanations. AI safety and trustworthiness need more attention, and this work makes the first step to explore the medical interpretability of LLMs.The dataset is available at https://github.com/HITsz-TMG/ExplainCPE.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑