多模态语言模型的性能研究
On the Performance of Multimodal Language Models
AI总结：
本文对比分析了不同多模态指令微调方法在复杂推理、对话、图像描述等任务上的性能，揭示了架构选择的关键见解，并指出当前方法在指令数据集多样性和响应真实性方面的不足。
摘要：
经过指令微调的大规模语言模型（LLMs）已在多种下游任务中展现出良好的零样本泛化能力。近期研究通过模型嫁接方式，将独立预训练的视觉编码器集成到LLMs中，从而为其引入多模态能力。这些多模态变体与LLMs一样经过指令微调，能够有效实现多模态任务的零样本泛化。本研究对不同的多模态指令微调方法进行了对比分析，并在一系列任务上评估其性能，包括复杂推理、对话、图像描述、多项选择题（MCQs）和二元分类。通过严格的基准测试和消融实验，我们揭示了在为LLMs引入多模态能力时指导架构选择的关键见解。然而，当前方法存在局限性；它们未能充分满足对多样化多模态指令数据集的需求，而这对增强任务泛化能力至关重要。此外，这些方法在生成响应时忽视了真实性和事实性方面的问题。这些发现阐明了当前在使语言模型适应图像理解方面的方法论限制，并为寻求利用多模态版本LLMs的研究人员和从业者提供了有价值的指导。
英文摘要：
Instruction-tuned large language models (LLMs) have demonstrated promising zero-shot generalization capabilities across various downstream tasks. Recent research has introduced multimodal capabilities to LLMs by integrating independently pretrained vision encoders through model grafting. These multimodal variants undergo instruction tuning, similar to LLMs, enabling effective zero-shot generalization for multimodal tasks. This study conducts a comparative analysis of different multimodal instruction tuning approaches and evaluates their performance across a range of tasks, including complex reasoning, conversation, image captioning, multiple-choice questions (MCQs), and binary classification. Through rigorous benchmarking and ablation experiments, we reveal key insights for guiding architectural choices when incorporating multimodal capabilities into LLMs. However, current approaches have limitations; they do not sufficiently address the need for a diverse multimodal instruction dataset, which is crucial for enhancing task generalization. Additionally, they overlook issues related to truthfulness and factuality when generating responses. These findings illuminate current methodological constraints in adapting language models for image comprehension and provide valuable guidance for researchers and practitioners seeking to harness multimodal versions of LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
