D-NLP 在 SemEval-2024 任务 2：评估大型语言模型的临床推理能力
D-NLP at SemEval-2024 Task 2: Evaluating Clinical Inference Capabilities of Large Language Models
AI总结：
本文以临床试验报告为数据集，评估主流开源与闭源 LLM 的自然语言推理能力，重点分析医学缩写和数值推理难题，Gemini 以 0.748 的 F1 排名第九。
摘要：
大型语言模型（LLM）因其在各类任务中的出色表现而获得了显著关注和广泛使用。然而，它们也并非没有自身的一系列挑战，包括幻觉、事实不一致以及数值—定量推理方面的限制等问题。在各种推理任务中评估 LLM 仍是一个活跃的研究领域。在 LLM 取得突破之前，Transformers 已经在医学领域证明了成功，被有效用于各种自然语言理解（NLU）任务。顺应这一趋势，LLM 也在医学领域得到训练和使用，这引发了关于事实准确性、对安全协议的遵守以及固有局限性的担忧。本文中，我们专注于使用临床试验报告作为数据集，评估流行的开源和闭源 LLM 的自然语言推理能力。我们展示了每个 LLM 的性能结果，并进一步分析它们在开发集上的表现，尤其关注涉及医学缩写以及需要数值—定量推理的困难实例。我们表现最佳的 LLM——Gemini 取得了 0.748 的测试集 F1 分数，在该任务排行榜上位列第九。我们的工作是此类研究中的首例，对 LLM 在医学领域内的推理能力进行了全面检验。
英文摘要：
Large language models (LLMs) have garnered significant attention and widespread usage due to their impressive performance in various tasks. However, they are not without their own set of challenges, including issues such as hallucinations, factual inconsistencies, and limitations in numerical-quantitative reasoning. Evaluating LLMs in miscellaneous reasoning tasks remains an active area of research. Prior to the breakthrough of LLMs, Transformers had already proven successful in the medical domain, effectively employed for various natural language understanding (NLU) tasks. Following this trend, LLMs have also been trained and utilized in the medical domain, raising concerns regarding factual accuracy, adherence to safety protocols, and inherent limitations. In this paper, we focus on evaluating the natural language inference capabilities of popular open-source and closed-source LLMs using clinical trial reports as the dataset. We present the performance results of each LLM and further analyze their performance on a development set, particularly focusing on challenging instances that involve medical abbreviations and require numerical-quantitative reasoning. Gemini, our leading LLM, achieved a test set F1-score of 0.748, securing the ninth position on the task scoreboard. Our work is the first of its kind, offering a thorough examination of the inference capabilities of LLMs within the medical domain.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
