Kosmos-2：将多模态大语言模型锚定到现实世界
Kosmos-2: Grounding Multimodal Large Language Models to the World
AI总结：
本文提出 Kosmos-2 多模态大语言模型，通过 Markdown 链接形式和 GrIT 数据实现文本与视觉对象的锚定，并在多类 grounding、指代及语言任务上验证其能力。
摘要：
我们推出了 Kosmos-2，这是一种多模态大语言模型（MLLM），具备感知对象描述（例如边界框）以及将文本锚定到视觉世界的新能力。具体而言，我们将指代表达表示为 Markdown 中的链接，即“[文本片段](边界框)”，其中对象描述是位置 token 序列。我们结合多模态语料库，构建了大规模的锚定图文对数据（称为 GrIT）来训练模型。除了 MLLM 的现有能力（例如感知通用模态、遵循指令以及执行上下文学习）之外，Kosmos-2 还将锚定能力整合到下游应用中。我们在广泛的任务上评估 Kosmos-2，包括（i）多模态锚定，如指代表达理解和短语锚定；（ii）多模态指代，如指代表达生成；（iii）感知-语言任务；以及（iv）语言理解与生成。这项工作为具身智能的发展奠定了基础，并揭示了语言、多模态感知、行动和世界建模的大融合，这是迈向通用人工智能的关键一步。代码和预训练模型可在 https://aka.ms/kosmos-2 获取。
英文摘要：
We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e.g., bounding boxes) and grounding text to the visual world. Specifically, we represent refer expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where object descriptions are sequences of location tokens. Together with multimodal corpora, we construct large-scale data of grounded image-text pairs (called GrIT) to train the model. In addition to the existing capabilities of MLLMs (e.g., perceiving general modalities, following instructions, and performing in-context learning), Kosmos-2 integrates the grounding capability into downstream applications. We evaluate Kosmos-2 on a wide range of tasks, including (i) multimodal grounding, such as referring expression comprehension, and phrase grounding, (ii) multimodal referring, such as referring expression generation, (iii) perception-language tasks, and (iv) language understanding and generation. This work lays out the foundation for the development of Embodiment AI and sheds light on the big convergence of language, multimodal perception, action, and world modeling, which is a key step toward artificial general intelligence. Code and pretrained models are available at https://aka.ms/kosmos-2.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
