arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.00876cs.CVcs.AIcs.LG
收藏

超越人类视觉:大型视觉语言模型在显微镜图像分析中的作用

Beyond Human Vision: The Role of Large Vision Language Models in Microscope Image Analysis

University of Arkansas(阿肯色大学)

Prateek Verma, Minh-Hao Van, Xintao Wu

AI总结:

本文系统评估了 ChatGPT、LLaVA、Gemini 和 SAM 在显微镜图像分类、分割、计数及视觉问答任务上的表现,发现其虽具备一定理解与分割能力,但仍远不及领域专家,易受杂质、缺陷和伪影干扰。

摘要:

视觉语言模型(VLMs)近年来崭露头角,因其能够理解图像与文本数据的双重模态而备受关注。LLaVA、ChatGPT-4 和 Gemini 等 VLM 最近在自然图像描述、视觉问答(VQA)和空间推理等任务上表现出令人印象深刻的性能。此外,Meta AI 提出的通用分割模型——Segment Anything Model(SAM)在从未见过的图像中隔离目标方面展现出前所未有的能力。由于医学专家、生物学家和材料科学家经常将显微镜或医学图像与说明文字、文献或报告形式的文本信息结合使用,并得出具有重大意义和价值的结论,因此在这些图像上测试 VLM 以及 SAM 等基础模型的性能无疑是至关重要的。在本研究中,我们让 ChatGPT、LLaVA、Gemini 和 SAM 在多种显微镜图像上执行分类、分割、计数和 VQA 任务。我们观察到,ChatGPT 和 Gemini 能够出色地理解显微镜图像中的视觉特征,而 SAM 在一般意义上相当擅长隔离图像中的伪影。然而,其性能与领域专家相比仍有差距——这些模型很容易受到图像中存在的杂质、缺陷、伪影重叠和多样性的干扰。

英文摘要:

Vision language models (VLMs) have recently emerged and gained the spotlight for their ability to comprehend the dual modality of image and textual data. VLMs such as LLaVA, ChatGPT-4, and Gemini have recently shown impressive performance on tasks such as natural image captioning, visual question answering (VQA), and spatial reasoning. Additionally, a universal segmentation model by Meta AI, Segment Anything Model (SAM) shows unprecedented performance at isolating objects from unforeseen images. Since medical experts, biologists, and materials scientists routinely examine microscopy or medical images in conjunction with textual information in the form of captions, literature, or reports, and draw conclusions of great importance and merit, it is indubitably essential to test the performance of VLMs and foundation models such as SAM, on these images. In this study, we charge ChatGPT, LLaVA, Gemini, and SAM with classification, segmentation, counting, and VQA tasks on a variety of microscopy images. We observe that ChatGPT and Gemini are impressively able to comprehend the visual features in microscopy images, while SAM is quite capable at isolating artefacts in a general sense. However, the performance is not close to that of a domain expert - the models are readily encumbered by the introduction of impurities, defects, artefact overlaps and diversity present in the images.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑