BuboGPT：在多模态大语言模型中实现视觉定位
BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs
AI总结：
本文提出 BuboGPT，通过基于 SAM 的视觉定位模块、两阶段训练与指令数据集，使多模态 LLM 具备文本、图像、音频联合理解及对象级定位能力。
摘要：
大语言模型（LLM）已经展现出通过语言与人类交互的卓越能力，尤其是在使用指令遵循数据的情况下。MiniGPT-4、LLaVA 和 X-LLM 等 LLM 的最新进展通过纳入图像、视频和语音等多模态输入，进一步扩展了其能力。尽管这些 LLM 能够针对给定模态信号生成精确而详细的语言理解，但它们放弃了定位输入中特定部分的能力，因此只能构建粗粒度映射。然而，文本与其他模态之间明确且信息丰富的对应关系不仅会改善用户体验，还有助于拓展多模态 LLM 的应用场景。因此，我们提出 BuboGPT，这是一种具备视觉定位能力的多模态 LLM，能够在视觉、音频和语言之间进行跨模态交互，提供对视觉对象及其他给定模态的细粒度理解。因此，BuboGPT 在为某个对象生成回答或描述时，能够指出该对象在图像中的具体位置。我们的贡献有两方面：1）一个基于 SAM 的现成视觉定位模块，可提取句子中的实体并在图像中找到相应掩码；2）一套两阶段训练方案和指令数据集，以赋予模型联合的文本-图像-音频理解能力。我们的实验表明，BuboGPT 在与人类交互过程中取得了令人瞩目的多模态理解和视觉定位能力。当提供任意模态组合（无论是否对齐）时，它都能持续表现良好。我们的代码、模型和数据集可在 https://bubo-gpt.github.io 获取。
英文摘要：
LLMs have demonstrated remarkable abilities at interacting with humans through language, especially with the usage of instruction-following data. Recent advancements in LLMs, such as MiniGPT-4, LLaVA, and X-LLM, further enlarge their abilities by incorporating multi-modal inputs, including image, video, and speech. Despite their effectiveness at generating precise and detailed language understanding of the given modality signal, these LLMs give up the ability to ground specific parts of inputs, thus only constructing a coarse-grained mapping. However, explicit and informative correspondence between text and other modalities will not only improve the user experience but also help to expand the application scenario of multi-modal LLMs. Therefore, we propose BuboGPT, a multi-modal LLM with visual grounding that can perform cross-modal interaction between vision, audio and language, providing fine-grained understanding of visual objects and other given modalities. As a result, BuboGPT is able to point out the specific location of an object in the image, when it is generating response or description for that object. Our contributions are two-fold: 1) An off-the-shelf visual grounding module based on SAM that extracts entities in a sentence and find corresponding masks in the image. 2) A two-stage training scheme and instruction dataset to endow joint text-image-audio understanding. Our experiments show that BuboGPT achieves impressive multi-modality understanding and visual grounding abilities during the interaction with human. It performs consistently well when provided by arbitrary modality combinations (either aligned or unaligned). Our code, model and dataset are available at https://bubo-gpt.github.io .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
