arXiv 2412.036652024-12-04cs.CVcs.AIcs.CLcs.MM

面向图像描述的多模态大语言模型个性化：一项实验分析

Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysis

University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）；IIT-CNR（意大利国家研究委员会信息科学与技术研究所）

Davide Bucciarelli, Nicholas Moratelli, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara

备注：: ECCV 2024 Workshop on Green Foundation Models

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文实验评估GPT-4V、Gemini等多模态LLM在图像描述基准上的零样本与提示学习、前缀微调、LoRA微调表现，发现其零样本能力强但领域适配与泛化难以兼顾。

摘要：

图像描述任务要求算法为视觉输入生成自然语言描述。近期进展显示，图像描述研究与大语言模型（LLM）及多模态LLM（如GPT-4V和Gemini）的发展正在融合，这些模型将纯文本LLM的能力扩展到多种模态。本文通过评估多模态LLM在各类图像描述基准上的性能，研究其能否取代传统图像描述网络。我们考察了这些模型的零样本能力，以及它们通过提示学习、前缀微调和低秩适应等微调方法适应不同语义领域的能力。结果表明，尽管多模态LLM取得了令人瞩目的零样本性能，但在针对特定领域进行微调的同时完整保持其泛化能力仍具挑战。我们讨论了这些发现对未来图像描述研究及开发适应性更强的多模态LLM的意义。

英文摘要：

The task of image captioning demands an algorithm to generate natural language descriptions of visual inputs. Recent advancements have seen a convergence between image captioning research and the development of Large Language Models (LLMs) and Multimodal LLMs -- like GPT-4V and Gemini -- which extend the capabilities of text-only LLMs to multiple modalities. This paper investigates whether Multimodal LLMs can supplant traditional image captioning networks by evaluating their performance on various image description benchmarks. We explore both the zero-shot capabilities of these models and their adaptability to different semantic domains through fine-tuning methods, including prompt learning, prefix tuning, and low-rank adaptation. Our results demonstrate that while Multimodal LLMs achieve impressive zero-shot performance, fine-tuning for specific domains while maintaining their generalization capabilities intact remains challenging. We discuss the implications of these findings for future research in image captioning and the development of more adaptable Multimodal LLMs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。