LMMs的黎明：GPT-4V(ision)的初步探索
The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)
AI总结：
本文初步探索GPT-4V(ision)的能力，通过精心设计的定性样本分析其处理交错多模态输入和视觉标记的通用性，并讨论应用场景与未来研究方向。
摘要：
大型多模态模型（LMMs）通过扩展大型语言模型（LLMs）的多感官能力（如视觉理解），实现了更强的通用智能。本文分析了最新模型GPT-4V(ision)，以加深对LMMs的理解。分析聚焦于GPT-4V能够执行的引人入胜的任务，包括用于探究GPT-4V能力质量和通用性的测试样本、其支持的输入和工作模式，以及有效的提示方法。在探索GPT-4V的过程中，我们策划并组织了一系列精心设计的定性样本，覆盖了多个领域和任务。这些样本的观察结果表明，GPT-4V在处理任意交错的多模态输入方面的空前能力及其能力的通用性，共同使GPT-4V成为一个强大的多模态通用系统。此外，GPT-4V理解输入图像上绘制的视觉标记的独特能力，催生了新的交互方法，如视觉引用提示。我们以对基于GPT-4V系统的新兴应用场景和未来研究方向的深入讨论作为报告的结论。我们希望这一初步探索能激发未来对下一代多模态任务表述、利用和增强LMMs解决实际问题的新方法，以及更好地理解多模态基础模型的研究。最后，我们承认所研究的模型完全是OpenAI创新工作的产物，其开发应完全归功于他们。关于作者和贡献归属，请参阅GPT-4V贡献论文：https://cdn.openai.com/contributions/gpt-4v.pdf
英文摘要：
Large multimodal models (LMMs) extend large language models (LLMs) with multi-sensory skills, such as visual understanding, to achieve stronger generic intelligence. In this paper, we analyze the latest model, GPT-4V(ision), to deepen the understanding of LMMs. The analysis focuses on the intriguing tasks that GPT-4V can perform, containing test samples to probe the quality and genericity of GPT-4V's capabilities, its supported inputs and working modes, and the effective ways to prompt the model. In our approach to exploring GPT-4V, we curate and organize a collection of carefully designed qualitative samples spanning a variety of domains and tasks. Observations from these samples demonstrate that GPT-4V's unprecedented ability in processing arbitrarily interleaved multimodal inputs and the genericity of its capabilities together make GPT-4V a powerful multimodal generalist system. Furthermore, GPT-4V's unique capability of understanding visual markers drawn on input images can give rise to new human-computer interaction methods such as visual referring prompting. We conclude the report with in-depth discussions on the emerging application scenarios and the future research directions for GPT-4V-based systems. We hope that this preliminary exploration will inspire future research on the next-generation multimodal task formulation, new ways to exploit and enhance LMMs to solve real-world problems, and gaining better understanding of multimodal foundation models. Finally, we acknowledge that the model under our study is solely the product of OpenAI's innovative work, and they should be fully credited for its development. Please see the GPT-4V contributions paper for the authorship and credit attribution: https://cdn.openai.com/contributions/gpt-4v.pdf
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
