揭示图像描述的有效上下文配置：外部与内部分析
Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis
AI总结：
本文针对图像描述任务，从外部示例配置策略和内部注意力特性分析双重视角系统研究大型多模态模型的上下文学习，揭示了配置对性能的影响及模型行为模式，并提出了可推广的新评估指标。
摘要：
大型模型的演化见证了上下文学习（ICL）能力的出现。在自然语言处理（NLP）中，大量研究已经证明了ICL的有效性。受大型语言模型（LLMs）成功的启发，研究人员开发了具备ICL能力的大型多模态模型（LMMs）。然而，针对多模态ICL的示例配置探索仍处于初步阶段。此外，上下文示例（ICEs）的可控性提供了一种高效且具有成本效益的手段，用于观察和分析LMMs在不同输入下的推理特性。本文针对图像描述任务，对多模态上下文学习进行了全面的外部和内部调查。在外部，我们通过三个维度探索示例配置策略：样本数量、图像检索和描述分配。我们采用多个指标来系统、全面地评估和总结关键发现。在内部，我们分析了典型LMM的注意力特性，并开发了基于注意力的指标来量化模型行为。我们还进行了辅助实验，以探索注意力驱动的模型加速和压缩的可行性。我们进一步比较了在相同模型设计和预训练策略下LMMs之间的性能差异，并从预训练数据特征的角度解释了这些差异。我们的研究通过外部实验揭示了ICEs配置策略如何影响模型性能，并通过内部检查揭示了典型的特性模式，为理解LMMs中的多模态ICL提供了双重视角。我们结合外部和内部分析来研究大型模型的方法，以及我们新提出的指标，可以应用于更广泛的研究领域。
英文摘要：
The evolution of large models has witnessed the emergence of In-Context Learning (ICL) capabilities. In Natural Language Processing (NLP), numerous studies have demonstrated the effectiveness of ICL. Inspired by the success of Large Language Models (LLMs), researchers have developed Large Multimodal Models (LMMs) with ICL capabilities. However, explorations of demonstration configuration for multimodal ICL remain preliminary. Additionally, the controllability of In-Context Examples (ICEs) provides an efficient and cost-effective means to observe and analyze the inference characteristics of LMMs under varying inputs. This paper conducts a comprehensive external and internal investigation of multimodal in-context learning on the image captioning task. Externally, we explore demonstration configuration strategies through three dimensions: shot number, image retrieval, and caption assignment. We employ multiple metrics to systematically and thoroughly evaluate and summarize key findings. Internally, we analyze typical LMM attention characteristics and develop attention-based metrics to quantify model behaviors. We also conduct auxiliary experiments to explore the feasibility of attention-driven model acceleration and compression. We further compare performance variations between LMMs with identical model design and pretraining strategies and explain the differences from the angles of pre-training data features. Our study reveals both how ICEs configuration strategies impact model performance through external experiments and characteristic typical patterns through internal inspection, providing dual perspectives for understanding multimodal ICL in LMMs. Our method of combining external and internal analysis to investigate large models, along with our newly proposed metrics, can be applied to broader research areas.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
