评估大语言模型的性能：来自中国药师资格考试的洞察
Assessing LLMs' Performance: Insights from the Chinese Pharmacist Exam
AI总结：
本研究比较了DeepSeek-R1和ChatGPT-4o在药师资格考试中的表现，发现DeepSeek-R1在准确性上显著优于后者，强调了领域特定模型在评估中的重要性及人类监督的必要性。
摘要：
背景：随着大语言模型（LLMs）越来越深入地融入数字健康教育和评估流程中，其在支持高风险、领域特定认证任务方面的能力仍缺乏深入研究。在中国，国家药师资格考试是评估药师临床和理论能力的标准基准。目标：本研究旨在比较两个大语言模型：ChatGPT-4o和DeepSeek-R1在2017-2021年中国药师资格考试（真实问题）上的表现，并讨论这些性能差异对AI赋能形成评估的启示。方法：从官方考试、培训材料和公共数据库中整理出共计2306道单选题（纯文本），排除包含表格或图像的问题。每个问题以原始中文格式输入，模型回答被评估为精确准确性。使用皮尔逊卡方检验比较总体表现，应用费舍尔精确检验分析年份间的单选题准确性。结果：DeepSeek-R1在总体准确性上显著优于ChatGPT-4o（90.0% vs. 76.1%，p < 0.001）。单元级分析显示DeepSeek-R1在基础和临床综合模块中具有持续优势。虽然年份间的单选题表现也更倾向于DeepSeek-R1，但这一表现差距在任何特定单元-年份中均未达到统计学显著性（所有p > 0.05）。结论：DeepSeek-R1在药师资格考试的结构和语义需求方面表现出稳健的契合度。这些发现表明，针对特定领域的模型值得进一步研究，同时也强调了在法律和伦理敏感情境中人类监督的必要性。
英文摘要：
Background: As large language models (LLMs) become increasingly integrated into digital health education and assessment workflows, their capabilities in supporting high-stakes, domain-specific certification tasks remain underexplored.In China, the national pharmacist licensure exam serves as a standardized benchmark for evaluating pharmacists' clinical and theoretical competencies. Objective: This study aimed to compare the performance of two LLMs: ChatGPT-4o and DeepSeek-R1 on real questions from the Chinese Pharmacist Licensing Examination (2017-2021), and to discuss the implications of these performance differences for AI-enabled formative evaluation. Methods: A total of 2,306 multiple-choice (text-only) questions were compiled from official exams, training materials, and public databases. Questions containing tables or images were excluded. Each item was input in its original Chinese format, and model responses were evaluated for exact accuracy. Pearson's Chi-squared test was used to compare overall performance, and Fisher's exact test was applied to year-wise multiple-choice accuracy. Results: DeepSeek-R1 outperformed ChatGPT-4o with a significantly higher overall accuracy (90.0% vs. 76.1%, p < 0.001). Unit-level analyses revealed consistent advantages for DeepSeek-R1, particularly in foundational and clinical synthesis modules. While year-by-year multiple-choice performance also favored DeepSeek-R1, this performance gap did not reach statistical significance in any specific unit-year (all p > 0.05). Conclusion: DeepSeek-R1 demonstrated robust alignment with the structural and semantic demands of the pharmacist licensure exam. These findings suggest that domain-specific models warrant further investigation for this context, while also reinforcing the necessity of human oversight in legally and ethically sensitive contexts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
