ChatSpot：通过精准指代指令调优自举多模态大语言模型
ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning
AI总结：
针对现有多模态大语言模型仅支持语言交互、准确性和效率受限的问题，本研究提出精准指代指令与支持多种交互形式的ChatSpot模型，构建多粒度指令跟随数据集，实验验证其在区域识别与交互上的优异性能。
摘要：
人机交互性是反映多模态大语言模型（MLLMs）可用性的关键维度。然而，现有端到端MLLMs仅支持用户通过语言指令与之交互，限制了交互的准确性与效率。本研究提出精准指代指令，采用点、框等多种参考表征作为指代提示，以指向特定区域，使MLLMs能够聚焦感兴趣区域，实现更细粒度的交互。基于精准指代指令，我们提出ChatSpot——一个统一的端到端多模态大语言模型，支持鼠标点击、拖拽、绘制框等多种交互形式，提供更灵活流畅的交互体验。我们还基于现有数据集和GPT-4生成，构建了一个多粒度视觉语言指令跟随数据集。此外，我们设计了一系列评估任务，以衡量区域识别与交互的有效性。实验结果展现了ChatSpot颇具前景的性能。
英文摘要：
Human-AI interactivity is a critical aspect that reflects the usability of multimodal large language models (MLLMs). However, existing end-to-end MLLMs only allow users to interact with them through language instructions, leading to the limitation of the interactive accuracy and efficiency. In this study, we present precise referring instructions that utilize diverse reference representations such as points and boxes as referring prompts to refer to the special region. This enables MLLMs to focus on the region of interest and achieve finer-grained interaction. Based on precise referring instruction, we propose ChatSpot, a unified end-to-end multimodal large language model that supports diverse forms of interactivity including mouse clicks, drag-and-drop, and drawing boxes, which provides a more flexible and seamless interactive experience. We also construct a multi-grained vision-language instruction-following dataset based on existing datasets and GPT-4 generating. Furthermore, we design a series of evaluation tasks to assess the effectiveness of region recognition and interaction. Experimental results showcase ChatSpot's promising performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
