TWIST & SCOUT：通过无遗忘调优为多模态大模型专家实现视觉 grounding
TWIST & SCOUT: Grounding Multimodal LLM-Experts by Forget-Free Tuning
AI总结：
针对现有多模态大语言模型缺乏空间感知能力且调优易遗忘原有技能的问题，提出TWIST & SCOUT框架，通过双专家调优模块和模拟人类推理的合成数据集实现无遗忘的视觉grounding能力增强，在多类基准任务上取得优异表现。
摘要：
空间感知能力是实现具身多模态AI系统的关键。然而，在缺乏大量空间监督数据的情况下，当前的多模态大语言模型（MLLMs）难以胜任该任务。本文提出TWIST & SCOUT框架，该框架可为预训练MLLMs赋予视觉grounding能力，同时不会遗忘其已有的图像与语言理解技能。为此，我们提出TWIST（双专家逐步调优模块），该模块通过一个在图像理解任务上预训练的冻结模块、以及一个用于视觉grounding任务的可学习模块来修改语言模型的解码器，使MLLM既能保留先前习得的知识与技能，又能获取缺失的能力。为实现模型的高效微调，我们生成了名为SCOUT的高质量合成数据集，该数据集模拟了人类在视觉grounding中的推理过程，提供了丰富的监督信号，描述了逐步的多模态推理流程，从而简化了视觉grounding任务。我们在多个标准基准数据集上评估了所提方法，涵盖grounded图像描述、零样本定位和视觉grounding任务。实验结果表明，我们的方法在所有任务上均表现出优异性能，同时保留了预训练的图像理解能力。
英文摘要：
Spatial awareness is key to enable embodied multimodal AI systems. Yet, without vast amounts of spatial supervision, current Multimodal Large Language Models (MLLMs) struggle at this task. In this paper, we introduce TWIST & SCOUT, a framework that equips pre-trained MLLMs with visual grounding ability without forgetting their existing image and language understanding skills. To this end, we propose TWIST, a twin-expert stepwise tuning module that modifies the decoder of the language model using one frozen module pre-trained on image understanding tasks and another learnable one for visual grounding tasks. This allows the MLLM to retain previously learned knowledge and skills, while acquiring what is missing. To fine-tune the model effectively, we generate a high-quality synthetic dataset we call SCOUT, which mimics human reasoning in visual grounding. This dataset provides rich supervision signals, describing a step-by-step multimodal reasoning process, thereby simplifying the task of visual grounding. We evaluate our approach on several standard benchmark datasets, encompassing grounded image captioning, zero-shot localization, and visual grounding tasks. Our method consistently delivers strong performance across all tasks, while retaining the pre-trained image understanding capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
