arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.07037cs.CVcs.AI
收藏

PathInsight:面向组织病理学智能辅助诊断的多模态数据集与模型的指令微调

PathInsight: Instruction Tuning of Multimodal Datasets and Models for Intelligence Assisted Diagnosis in Histopathology

College of Engineering, Peking University(北京大学工学院)

Xiaomin Wu, Rui Xu, Pengchen Wei, Wenkang Qin, Peixiang Huang, Ziheng Li, Lin Luo

AI总结:

针对病理诊断中多模态大模型训练部署成本高、优质数据集稀缺的问题,本研究构建了含约4.5万病例、覆盖6类以上任务的数据集,微调LLaVA等多模态大模型,验证了其在病理任务上的优异表现,并公开资源助力相关研究。

摘要:

病理诊断仍是肿瘤识别的 definitive 标准。多模态大模型的兴起简化了图像分析与文本描述的整合流程。尽管取得了这一进展,训练和部署这些复杂多模态模型的高昂成本,加上高质量训练数据集的稀缺,在前沿技术与其临床场景应用之间造成了显著鸿沟。我们精心构建了一个约45000个病例的数据集,覆盖6种以上不同任务,包括器官组织分类、病理报告描述生成,以及病理相关问答问题的解答。我们使用该数据集微调了多模态大模型,具体为LLaVA、Qwen-VL、InternLM,以提升其基于指令的性能。我们对基础模型和微调模型在特定数据集上执行图像描述和分类任务的能力进行了定性评估。评估结果表明,微调后的模型在解答典型病理问题方面表现出色。我们希望通过公开我们的模型和数据集,为医学界和研究界提供有价值的资源。

英文摘要:

Pathological diagnosis remains the definitive standard for identifying tumors. The rise of multimodal large models has simplified the process of integrating image analysis with textual descriptions. Despite this advancement, the substantial costs associated with training and deploying these complex multimodal models, together with a scarcity of high-quality training datasets, create a significant divide between cutting-edge technology and its application in the clinical setting. We had meticulously compiled a dataset of approximately 45,000 cases, covering over 6 different tasks, including the classification of organ tissues, generating pathology report descriptions, and addressing pathology-related questions and answers. We have fine-tuned multimodal large models, specifically LLaVA, Qwen-VL, InternLM, with this dataset to enhance instruction-based performance. We conducted a qualitative assessment of the capabilities of the base model and the fine-tuned model in performing image captioning and classification tasks on the specific dataset. The evaluation results demonstrate that the fine-tuned model exhibits proficiency in addressing typical pathological questions. We hope that by making both our models and datasets publicly available, they can be valuable to the medical and research communities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑