开源大语言模型在医疗文本分类任务上的基准评测
Benchmarking Open-Source Large Language Models on Healthcare Text Classification Tasks
AI总结：
本文对五个开源大语言模型在六项医疗文本分类任务上进行基准评测，发现 DeepSeekV3 整体表现最佳，并指出模型选择应依据具体任务领域和精确率-召回率需求而非仅看参数量。
摘要：
大语言模型（LLMs）在医疗信息抽取中的应用已成为一种有前景的方法。本研究评估了五个开源大语言模型的分类性能：GEMMA-3-27B-IT、LLAMA3-70B、LLAMA4-109B、DEEPSEEK-R1-DISTILL-LLAMA-70B 和 DEEPSEEK-V3-0324-UD-Q2_K_XL，涵盖六项医疗相关分类任务，涉及社交媒体数据（乳腺癌、用药方案变更、不良妊娠结局、潜在 COVID-19 病例）和临床数据（污名标注、用药变更讨论）。我们报告了所有模型-任务组合的精确率、召回率和 F1 分数，并附有 95% 置信区间。研究结果揭示了各 LLM 之间存在显著的性能差异，其中 DeepSeekV3 成为整体表现最强的模型，在四项任务中取得了最高的 F1 分数。值得注意的是，模型在社交媒体任务上的表现普遍优于临床数据任务，这表明可能存在领域特定的挑战。GEMMA-3-27B-IT 尽管参数量较小，却表现出极高的召回率；而 LLAMA4-109B 与其前代 LLAMA3-70B 相比表现意外地不尽如人意，表明更大的参数量并不能保证更好的分类结果。我们观察到不同模型之间存在明显的精确率-召回率权衡，有些模型偏向敏感性而非特异性，反之亦然。这些发现凸显了在医疗应用中针对特定任务进行模型选择的重要性，需要同时考虑具体的数据领域和精确率-召回率需求，而不仅仅是模型规模。随着医疗领域越来越多地整合 AI 驱动的文本分类工具，这项全面的基准评测为模型选择与实施提供了有价值的指导，同时也强调了在医疗场景中持续评估和领域适配 LLMs 的必要性。
英文摘要：
The application of large language models (LLMs) to healthcare information extraction has emerged as a promising approach. This study evaluates the classification performance of five open-source LLMs: GEMMA-3-27B-IT, LLAMA3-70B, LLAMA4-109B, DEEPSEEK-R1-DISTILL-LLAMA-70B, and DEEPSEEK-V3-0324-UD-Q2_K_XL, across six healthcare-related classification tasks involving both social media data (breast cancer, changes in medication regimen, adverse pregnancy outcomes, potential COVID-19 cases) and clinical data (stigma labeling, medication change discussion). We report precision, recall, and F1 scores with 95% confidence intervals for all model-task combinations. Our findings reveal significant performance variability between LLMs, with DeepSeekV3 emerging as the strongest overall performer, achieving the highest F1 scores in four tasks. Notably, models generally performed better on social media tasks compared to clinical data tasks, suggesting potential domain-specific challenges. GEMMA-3-27B-IT demonstrated exceptionally high recall despite its smaller parameter count, while LLAMA4-109B showed surprisingly underwhelming performance compared to its predecessor LLAMA3-70B, indicating that larger parameter counts do not guarantee improved classification results. We observed distinct precision-recall trade-offs across models, with some favoring sensitivity over specificity and vice versa. These findings highlight the importance of task-specific model selection for healthcare applications, considering the particular data domain and precision-recall requirements rather than model size alone. As healthcare increasingly integrates AI-driven text classification tools, this comprehensive benchmarking provides valuable guidance for model selection and implementation while underscoring the need for continued evaluation and domain adaptation of LLMs in healthcare contexts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
