MoColl：基于智能体的特定与通用模型协作的图像描述生成
MoColl: Agent-Based Specific and General Model Collaboration for Image Captioning
AI总结：
本文提出智能体增强的模型协作框架MoColl，通过将图像描述任务分解为问答子任务，结合可训练VQA模型与基于LLM的智能体，有效整合特定领域与通用知识，显著提升了放射学报告生成质量。
摘要：
图像描述生成是计算机视觉与自然语言处理交叉领域的一项关键任务，在各个领域有着广泛的应用。对于诊断报告生成等复杂任务，深度学习模型不仅需要特定领域的图像-描述数据集，还需要结合相关的通用知识以提供上下文准确性。现有方法存在固有局限性：专用模型擅长捕捉特定领域细节但缺乏泛化能力，而基于大型语言模型（LLMs）构建的视觉语言模型（VLMs）虽能利用通用知识，却难以适应特定领域。为解决这些局限性，本文提出了一种新颖的智能体增强模型协作框架，称为MoColl，旨在有效整合特定领域与通用知识。具体而言，我们的方法是将复杂的图像描述生成任务分解为一系列相互关联的问答子任务。采用一个可训练的视觉问答（VQA）模型作为专用工具，专注于特定领域的视觉分析，根据图像内容回答任务特定的问题。同时，一个基于LLM且具备通用知识的智能体负责提出这些问题，并将生成的问答对综合成连贯的描述。除了利用VQA模型外，该智能体还进一步指导其训练，以增强其特定领域能力。在放射学报告生成上的实验结果验证了所提框架的有效性，表明生成报告的质量有显著提升。
英文摘要：
Image captioning is a critical task at the intersection of computer vision and natural language processing, with wide-ranging applications across various domains. For complex tasks such as diagnostic report generation, deep learning models require not only domain-specific image-caption datasets but also the incorporation of relevant general knowledge to provide contextual accuracy. Existing approaches exhibit inherent limitations: specialized models excel in capturing domain-specific details but lack generalization, while vision-language models (VLMs) built on large language models (LLMs) leverage general knowledge but struggle with domain-specific adaptation. To address these limitations, this paper proposes a novel agent-enhanced model collaboration framework, which we call MoColl, designed to effectively integrate domain-specific and general knowledge. Specifically, our approach is to decompose complex image captioning tasks into a series of interconnected question-answer subtasks. A trainable visual question answering (VQA) model is employed as a specialized tool to focus on domain-specific visual analysis, answering task-specific questions based on image content. Concurrently, an LLM-based agent with general knowledge formulates these questions and synthesizes the resulting question-answer pairs into coherent captions. Beyond its role in leveraging the VQA model, the agent further guides its training to enhance its domain-specific capabilities. Experimental results on radiology report generation validate the effectiveness of the proposed framework, demonstrating significant improvements in the quality of generated reports.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
