Shikra：释放多模态大语言模型的指称对话魔力
Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
AI总结：
本文提出多模态大语言模型Shikra，以自然语言直接处理空间坐标输入输出，无需额外模块即可完成指称对话及多种视觉-语言任务，并支持坐标推理等新应用。
摘要：
在人类对话中，个体可以在与他人交流时指出场景中的相关区域，而对方在必要时也可以通过引用特定区域进行回应。这种自然的指称对话能力在当前的多模态大语言模型（MLLMs）中仍然缺失。为填补这一空白，本文提出了一种名为Shikra的MLLM，它能够以自然语言处理空间坐标输入和输出。其架构由视觉编码器、对齐层和大语言模型（LLM）组成，设计上力求直接和简单，无需额外的词汇表、位置编码器、检测前/后模块或外部插件模型，所有输入和输出均采用自然语言形式。指称对话是多种视觉-语言（VL）任务的超集。Shikra能够自然处理诸如指称表达理解（REC）和点定位问答（PointQA）等与位置相关的任务，以及图像描述和视觉问答（VQA）等传统VL任务。实验结果展示了Shikra令人鼓舞的性能。此外，它还支持众多令人兴奋的应用，例如在思维链中提供所提及物体的坐标，以及比较用户所指区域的相似性。我们的代码、模型和数据集可在https://github.com/shikras/shikra获取。
英文摘要：
In human conversations, individuals can indicate relevant regions within a scene while addressing others. In turn, the other person can then respond by referring to specific regions if necessary. This natural referential ability in dialogue remains absent in current Multimodal Large Language Models (MLLMs). To fill this gap, this paper proposes an MLLM called Shikra, which can handle spatial coordinate inputs and outputs in natural language. Its architecture consists of a vision encoder, an alignment layer, and a LLM. It is designed to be straightforward and simple, without the need for extra vocabularies, position encoder, pre-/post-detection modules, or external plug-in models. All inputs and outputs are in natural language form. Referential dialogue is a superset of various vision-language (VL) tasks. Shikra can naturally handle location-related tasks like REC and PointQA, as well as conventional VL tasks such as Image Captioning and VQA. Experimental results showcase Shikra's promising performance. Furthermore, it enables numerous exciting applications, like providing mentioned objects' coordinates in chains of thoughts and comparing user-pointed regions similarities. Our code, model and dataset are accessed at https://github.com/shikras/shikra.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
