arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.12547cs.CL
收藏

迈向评估与构建医学领域多功能大语言模型

Towards Evaluating and Building Versatile Large Language Models for Medicine

Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);China Mobile Communications Group Co., Ltd.(中国移动通信集团有限公司);China Mobile Communications Group Shanghai Co., Ltd.(中国移动通信集团上海有限公司)

Chaoyi Wu, Pengcheng Qiu, Jinxin Liu, Hongfei Gu, Na Li, Ya Zhang, Yanfeng Wang, Weidi Xie

AI总结:

本文提出医学综合基准 MedS-Bench 和大规模指令微调数据集 MedS-Ins,并通过微调得到 MMedIns-Llama 3,显著提升大语言模型在多种临床任务上的表现。

摘要:

在本研究中,我们提出了 MedS-Bench,一个旨在评估大语言模型(LLMs)在临床场景中表现的综合基准。与现有侧重于多项选择题问答的基准不同,MedS-Bench 涵盖 11 个高层级临床任务,包括临床报告摘要、治疗建议、诊断、命名实体识别和医学概念解释等。我们使用少样本提示评估了六种领先的大语言模型,例如 MEDITRON、Mistral、InternLM 2、Llama 3、GPT-4 和 Claude-3.5,发现即使是最先进的模型在这些复杂任务上也表现不佳。为了解决这些局限,我们构建了 MedS-Ins,一个面向医学的大规模指令微调数据集。MedS-Ins 包含 58 个医学导向的语言语料库,总计 1350 万个样本,覆盖 122 个任务。为证明该数据集的实用性,我们进行了一项概念验证实验,对一个轻量级开源医学语言模型进行指令微调。所得模型 MMedIns-Llama 3 在几乎所有临床任务上都显著优于现有模型。为促进大语言模型在临床挑战中应用的进一步发展,我们已将 MedS-Ins 数据集完全开放,并邀请研究社区为其扩展做出贡献。此外,我们还推出了 MedS-Bench 的动态排行榜,计划定期更新测试集以跟踪进展,并增强通用大语言模型对医学领域的适应能力。排行榜:https://henrychur.github.io/MedS-Bench/。Github:https://github.com/MAGIC-AI4Med/MedS-Ins。

英文摘要:

In this study, we present MedS-Bench, a comprehensive benchmark designed to evaluate the performance of large language models (LLMs) in clinical contexts. Unlike existing benchmarks that focus on multiple-choice question answering, MedS-Bench spans 11 high-level clinical tasks, including clinical report summarization, treatment recommendations, diagnosis, named entity recognition, and medical concept explanation, among others. We evaluated six leading LLMs, e.g., MEDITRON, Mistral, InternLM 2, Llama 3, GPT-4, and Claude-3.5 using few-shot prompting, and found that even the most sophisticated models struggle with these complex tasks. To address these limitations, we developed MedS-Ins, a large-scale instruction tuning dataset for medicine. MedS-Ins comprises 58 medically oriented language corpora, totaling 13.5 million samples across 122 tasks. To demonstrate the dataset's utility, we conducted a proof-of-concept experiment by performing instruction tuning on a lightweight, open-source medical language model. The resulting model, MMedIns-Llama 3, significantly outperformed existing models across nearly all clinical tasks. To promote further advancements in the application of LLMs to clinical challenges, we have made the MedS-Ins dataset fully accessible and invite the research community to contribute to its expansion.Additionally, we have launched a dynamic leaderboard for MedS-Bench, which we plan to regularly update the test set to track progress and enhance the adaptation of general LLMs to the medical domain. Leaderboard: https://henrychur.github.io/MedS-Bench/. Github: https://github.com/MAGIC-AI4Med/MedS-Ins.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑