MM-SeR: 多模态自反思用于轻量级图像描述
MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioning
AI总结：
MM-SeR通过轻量级多模态自反思框架，在减少计算成本的同时，实现了与大规模模型相当的图像描述性能，并扩展到长距离视频问答任务。
摘要：
视频聊天机器人和导航机器人等系统往往依赖于流式图像描述来解读视觉输入。现有方法通常使用大规模多模态语言模型（MLLMs）来完成此任务，但其巨大的计算成本阻碍了实际应用。这一限制促使我们开发了一个轻量级描述模型。我们的研究首先通过将MLLMs中的大规模语言组件替换为一个参数仅为12500万的紧凑模型来开始。令人惊讶的是，尽管模型大小减少了93倍，该紧凑模型在性能上与MLLMs相当，这表明事实性的图像描述并不显著需要LLM的复杂推理能力。尽管这一有希望的结果，我们的轻量模型仍然缺乏可靠性。为了解决这个问题，我们从人类视觉过程获得灵感：在关注细节之前，先获得场景的整体和粗略理解。因此，我们提出了一种多模态自反思框架，引导模型利用通过参考先前粗略描述所识别的显著区域的特征，并生成更精确的描述。实验结果表明，我们的模型在单句和详细描述方面都表现出色，甚至延伸到长距离视频问答任务。
英文摘要：
Systems such as video chatbots and navigation robots often depend on streaming image captioning to interpret visual inputs. Existing approaches typically employ large multimodal language models (MLLMs) for this purpose, but their substantial computational cost hinders practical application. This limitation motivates our development of a lightweight captioning model. Our investigation begins by replacing the large-scale language component in MLLMs with a compact 125M-parameter model. Surprisingly, this compact model, despite a 93x reduction in size, achieves comparable performance to MLLMs, suggesting that factual image captioning does not significantly require the complex reasoning abilities of LLMs. Despite this promising result, our lightweight model still lacks reliability. To address this, we draw inspiration from the human visual process: perceiving a global and coarse understanding of the scene before attending to finer details. Accordingly, we propose a multimodal self-refinement framework that guides the model to utilize features from salient regions, identified by referencing the previous coarse caption, and to produce a refined description. Experimental results demonstrate the superiority of our model in both single-sentence and detailed captioning, extending even to long-range video QA tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
