住院医师考试中正确答案的解释性论证抽取
Explanatory Argument Extraction of Correct Answers in Resident Medical Exams
AI总结：
本文提出一个基于西班牙住院医师考试的新数据集和抽取式任务，包含正确答案及错误答案的解释性论证，并利用抽取式问答自动评估大语言模型，实验表明多语言模型有时优于单语及医学领域适配模型。
摘要：
开发所需技术以辅助医学专家开展日常活动，是当前人工智能研究领域的热点课题。因此，近期提出了许多大语言模型（LLMs）和自动化基准，旨在利用自然语言作为人机交互的中介工具，促进循证医学（EBM）中的信息抽取。最具代表性的基准仅限于多项选择题或长文本答案，并且仅提供英文版本。为了解决这些不足，本文提出了一个新的数据集，与以往工作不同，该数据集：（i）不仅包含正确答案的解释性论证，还包含用于推理错误答案为何不正确的论证；（ii）这些解释由医学医生原创撰写，用于回答西班牙住院医师医学考试中的问题。此外，这一新基准使我们能够构建一个新颖的抽取式任务，即识别由医学医生撰写的正确答案解释。我们设置的另一个优势在于，可以利用抽取式问答范式自动评估大语言模型的性能，而无需依赖医学专家进行昂贵的人工评估。针对西班牙语语言模型的综合实验表明，有时多语言模型的表现优于单语模型，甚至超过了已适配医学领域的模型。此外，单语模型之间的结果表现不一，被认为规模更小、性能更差的模型反而具有竞争力。无论如何，所获得的结果表明，我们的新数据集和方法可以成为一种有效的技术，帮助医学从业者识别与医学问题相关的循证解释。
英文摘要：
Developing the required technology to assist medical experts in their everyday activities is currently a hot topic in the Artificial Intelligence research field. Thus, a number of large language models (LLMs) and automated benchmarks have recently been proposed with the aim of facilitating information extraction in Evidence-Based Medicine (EBM) using natural language as a tool for mediating in human-AI interaction. The most representative benchmarks are limited to either multiple-choice or long-form answers and are available only in English. In order to address these shortcomings, in this paper we present a new dataset which, unlike previous work: (i) includes not only explanatory arguments for the correct answer, but also arguments to reason why the incorrect answers are not correct; (ii) the explanations are written originally by medical doctors to answer questions from the Spanish Residency Medical Exams. Furthermore, this new benchmark allows us to setup a novel extractive task which consists of identifying the explanation of the correct answer written by medical doctors. An additional benefit of our setting is that we can leverage the extractive QA paradigm to automatically evaluate performance of LLMs without resorting to costly manual evaluation by medical experts. Comprehensive experimentation with language models for Spanish shows that sometimes multilingual models fare better than monolingual ones, even outperforming models which have been adapted to the medical domain. Furthermore, results across the monolingual models are mixed, with supposedly smaller and inferior models performing competitively. In any case, the obtained results show that our novel dataset and approach can be an effective technique to help medical practitioners in identifying relevant evidence-based explanations for medical questions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
