arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.17484cs.CL
收藏

MedCare:通过解耦临床对齐与知识聚合来推进医疗大语言模型

MedCare: Advancing Medical LLMs through Decoupling Clinical Alignment and Knowledge Aggregation

Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)

Yusheng Liao, Shuyang Jiang, Zhe Chen, Yanfeng Wang, Yu Wang

AI总结:

针对医疗任务复杂多样导致泛化性差的问题,提出渐进式两阶段微调流水线MedCare,通过解耦临床对齐与知识聚合缓解知识遗忘,在20多项医疗任务上实现SOTA且各规模模型均显著提升。

摘要:

大语言模型(LLMs)在自然语言理解与生成方面取得了显著进展,在医疗领域尤为具有价值。尽管取得进步,但医疗任务固有的复杂性与多样性带来了持续挑战,这些任务可被分为知识密集型任务和对齐需求型任务。以往方法要么忽略后者,要么仅关注少数任务,从而丧失泛化能力。为解决这些缺陷,我们提出了一种渐进式微调流水线。该流水线在第一阶段采用知识聚合器和噪声聚合器来编码多样化知识并过滤有害信息。在第二阶段,我们丢弃噪声聚合器以避免次优表示的干扰,并利用一个沿知识空间正交方向优化的额外对齐模块来缓解知识遗忘。基于这一两阶段范式,我们通过解耦临床对齐与知识聚合提出了医疗LLM(MedCare),旨在20多项医疗任务及特定医疗对齐任务上实现最先进(SOTA)性能。不同规模的MedCare(1.8B、7B、14B)相比同等规模的现有模型均展现出显著提升。

英文摘要:

Large language models (LLMs) have shown substantial progress in natural language understanding and generation, proving valuable especially in the medical field. Despite advancements, challenges persist due to the complexity and diversity inherent in medical tasks, which can be categorized as knowledge-intensive tasks and alignment-required tasks. Previous approaches either ignore the latter task or focus on a minority of tasks and hence lose generalization. To address these drawbacks, we propose a progressive fine-tuning pipeline. This pipeline employs a Knowledge Aggregator and a Noise aggregator to encode diverse knowledge in the first stage and filter out detrimental information. In the second stage, we drop the Noise Aggregator to avoid the interference of suboptimal representation and leverage an additional alignment module optimized towards an orthogonal direction to the knowledge space to mitigate knowledge forgetting. Based on this two-stage paradigm, we proposed a Medical LLM through decoupling Clinical Alignment and Knowledge Aggregation (MedCare), which is designed to achieve state-of-the-art (SOTA) performance on over 20 medical tasks, as well as SOTA results on specific medical alignment tasks. Various model sizes of MedCare (1.8B, 7B, 14B) all demonstrate significant improvements over existing models with similar model sizes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑