KorMedMCQA：面向韩国医疗执业资格考试的多项选择题问答基准
KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinations
AI总结：
本研究推出首个韩国医学多项选择问答基准KorMedMCQA，评估59个大语言模型性能，发现思维链推理可提升性能，且MedQA无法可靠替代该基准评估韩国医疗AI模型，已公开发布数据集。
摘要：
我们推出了KorMedMCQA，这是首个韩国医学多项选择题问答基准，其数据来源于2012年至2024年间韩国举办的专业医疗执业资格考试。该数据集包含7469道来自医师、护士、药师和牙医考试的题目，覆盖广泛的医学学科。我们评估了59个大语言模型的性能，涵盖专有模型与开源模型、多语言模型与韩语专用模型，以及针对临床应用微调的模型。结果表明，与直接作答方法相比，应用思维链（Chain of Thought, CoT）推理最多可将模型性能提升4.5%。我们还探究了MedQA（源自美国执业医师资格考试、应用最广泛的医学基准之一）能否作为评估其他地区（本研究中为韩国）模型性能的可靠替代指标。我们对模型在KorMedMCQA与MedQA上的得分进行相关性分析，发现这两个基准的一致性并不比完全不同领域的基准（例如MedQA与MMLU-Pro）更好。这一发现凸显了韩国与美国医疗语境在语言和临床方面的显著差异，进一步证明了构建区域专属医学问答基准的必要性。为支持韩国医疗AI领域的持续研究，我们通过Huggingface公开发布了KorMedMCQA。
英文摘要：
We present KorMedMCQA, the first Korean Medical Multiple-Choice Question Answering benchmark, derived from professional healthcare licensing examinations conducted in Korea between 2012 and 2024. The dataset contains 7,469 questions from examinations for doctor, nurse, pharmacist, and dentist, covering a wide range of medical disciplines. We evaluate the performance of 59 large language models, spanning proprietary and open-source models, multilingual and Korean-specialized models, and those fine-tuned for clinical applications. Our results show that applying Chain of Thought (CoT) reasoning can enhance the model performance by up to 4.5% compared to direct answering approaches. We also investigate whether MedQA, one of the most widely used medical benchmarks derived from the U.S. Medical Licensing Examination, can serve as a reliable proxy for evaluating model performance in other regions-in this case, Korea. Our correlation analysis between model scores on KorMedMCQA and MedQA reveals that these two benchmarks align no better than benchmarks from entirely different domains (e.g., MedQA and MMLU-Pro). This finding underscores the substantial linguistic and clinical differences between Korean and U.S. medical contexts, reinforcing the need for region-specific medical QA benchmarks. To support ongoing research in Korean healthcare AI, we publicly release the KorMedMCQA via Huggingface.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
