扩展大型视觉语言模型以增强生物医学图像分析中的多模态理解
Scaling Large Vision-Language Models for Enhanced Multimodal Comprehension In Biomedical Image Analysis
AI总结：
本研究基于LLaVA模型微调智能助手，利用42,673篇文章和50,882个图文对训练，显著提升了低剂量放射治疗领域视觉问答任务的多模态理解能力，有效减少幻觉并增强特定领域理解。
摘要：
大型语言模型（LLMs）在理解文本数据方面展现了巨大能力，并日益被用于通过知识提取（信息检索）、知识蒸馏（将关键发现和方法总结为简明形式）和知识综合（聚合多个科学来源的信息以解决复杂查询、生成假设和制定实验计划）来帮助研究人员加速科学发现。然而，科学数据通常以视觉和文本两种模态存在。视觉语言模型（VLMs）通过引入预训练视觉骨干网络处理图像，并使用跨模态投影器将图像令牌适配到LLM维度空间来解决这一问题，从而提供更丰富的多模态理解。尽管如此，现成的VLMs在处理特定领域数据时能力有限，且易产生幻觉。我们开发了基于LLaVA模型微调的智能助手，以增强低剂量放射治疗（LDRT，一种用于治疗癌症相关疾病的良性方法）中的多模态理解。利用来自42,673篇文章的多语言数据，我们为视觉问答（VQA）基准设计了复杂推理和详细描述任务。我们的助手在50,882个图文对上训练，使用LLM作为评判者的方法评估，表现出优于基础模型的性能，特别是在减少幻觉和提升特定领域理解方面。
英文摘要：
Large language models (LLMs) have demonstrated immense capabilities in understanding textual data and are increasingly being adopted to help researchers accelerate scientific discovery through knowledge extraction (information retrieval), knowledge distillation (summarizing key findings and methodologies into concise forms), and knowledge synthesis (aggregating information from multiple scientific sources to address complex queries, generate hypothesis and formulate experimental plans). However, scientific data often exists in both visual and textual modalities. Vision language models (VLMs) address this by incorporating a pretrained vision backbone for processing images and a cross-modal projector that adapts image tokens into the LLM dimensional space, thereby providing richer multimodal comprehension. Nevertheless, off-the-shelf VLMs show limited capabilities in handling domain-specific data and are prone to hallucinations. We developed intelligent assistants finetuned from LLaVA models to enhance multimodal understanding in low-dose radiation therapy (LDRT)-a benign approach used in the treatment of cancer-related illnesses. Using multilingual data from 42,673 articles, we devise complex reasoning and detailed description tasks for visual question answering (VQA) benchmarks. Our assistants, trained on 50,882 image-text pairs, demonstrate superior performance over base models as evaluated using LLM-as-a-judge approach, particularly in reducing hallucination and improving domain-specific comprehension.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
