ChatterBox：多轮多模态指代与定位
ChatterBox: Multi-round Multimodal Referring and Grounding
AI总结：
本文提出多模态多轮指代与定位任务及 CB-300K 基准，并构建双分支 ChatterBox 模型，结合区域标记化、查询定位与两阶段优化，在定量和定性实验中优于现有模型。
摘要：
在本研究中，我们为一项名为多模态多轮指代与定位（MRG）的新任务建立了基线，为实例级多模态对话开辟了一个有前景的方向。为此，我们提出了一个新的基准和一个高效的视觉-语言模型。名为 CB-300K 的新基准涵盖了多轮对话、多个实例之间复杂的空间关系以及一致性推理等挑战，这些挑战超出了现有基准所呈现的范围。所提出的模型名为 ChatterBox，采用双分支架构协同处理视觉和语言任务。通过将实例区域标记化，语言分支获得了感知指称信息的能力。同时，ChatterBox 将视觉分支中的查询嵌入馈送至 token 接收器以进行视觉定位。我们设计了两阶段优化策略，同时利用 CB-300K 和辅助外部数据来提高模型在实例级理解方面的稳定性和能力。实验表明，ChatterBox 在 MRG 的定量和定性表现上均优于现有模型，为实现复杂而精确交互的多模态对话场景铺平了新道路。代码、数据和模型可在 https://github.com/sunsmarterjie/ChatterBox 获取。
英文摘要：
In this study, we establish a baseline for a new task named multimodal multi-round referring and grounding (MRG), opening up a promising direction for instance-level multimodal dialogues. We present a new benchmark and an efficient vision-language model for this purpose. The new benchmark, named CB-300K, spans challenges including multi-round dialogue, complex spatial relationships among multiple instances, and consistent reasoning, which are beyond those shown in existing benchmarks. The proposed model, named ChatterBox, utilizes a two-branch architecture to collaboratively handle vision and language tasks. By tokenizing instance regions, the language branch acquires the ability to perceive referential information. Meanwhile, ChatterBox feeds a query embedding in the vision branch to a token receiver for visual grounding. A two-stage optimization strategy is devised, making use of both CB-300K and auxiliary external data to improve the model's stability and capacity for instance-level understanding. Experiments show that ChatterBox outperforms existing models in MRG both quantitatively and qualitatively, paving a new path towards multimodal dialogue scenarios with complicated and precise interactions. Code, data, and model are available at: https://github.com/sunsmarterjie/ChatterBox.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
