arXiv 2312.124362023-12-20cs.CVcs.AIcs.CLcs.MM

GPT-4V的挑战者？Gemini在视觉专业知识上的早期探索

A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise

Tencent Youtu Lab（腾讯优图实验室）；Shanghai AI Laboratory（上海人工智能实验室）；CUHK MMLab（香港中文大学多媒体实验室）；ECNU（华东师范大学）；USTC（中国科学技术大学）；Peking University（北京大学）

Chaoyou Fu, Renrui Zhang, Zihan Wang, Yubo Huang, Zhengye Zhang, Longtian Qiu, Gaoxiang Ye, Yunhang Shen, Mengdan Zhang, Peixian Chen, Sirui Zhao, Shaohui Lin, … 展开作者

Chaoyou Fu, Renrui Zhang, Zihan Wang, Yubo Huang, Zhengye Zhang, Longtian Qiu, Gaoxiang Ye, Yunhang Shen, Mengdan Zhang, Peixian Chen, Sirui Zhao, Shaohui Lin, Deqiang Jiang, Di Yin, Peng Gao, Ke Li, Hongsheng Li, Xing Sun

备注：: Total 120 pages. See our project at https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文初步探索Gemini Pro的视觉理解能力，与GPT-4V和Sphinx对比，发现Gemini在多个领域可与GPT-4V匹敌，但仍有差距，并指出MLLM的常见问题。

摘要：

多模态大语言模型（MLLMs）的兴趣激增，例如OpenAI的GPT-4V（ision），已在学术界和工业界标志着显著趋势。它们赋予大语言模型（LLMs）强大的视觉理解能力，使其能够处理多样的多模态任务。最近，Google发布了Gemini，这是其最新且最强大的MLLM，从头构建以支持多模态。鉴于其卓越的推理能力，Gemini能否挑战GPT-4V在多模态学习中的领先地位？在本文中，我们对Gemini Pro的视觉理解能力进行了初步探索，全面涵盖四个领域：基础感知、高级认知、挑战性视觉任务以及各种专家能力。我们将Gemini Pro与最先进的GPT-4V进行比较以评估其上限，并与最新的开源MLLM Sphinx进行比较，这揭示了人工努力与黑盒系统之间的差距。定性样本表明，虽然GPT-4V和Gemini展示了不同的回答风格和偏好，但它们可以表现出相当的视觉推理能力，而Sphinx在领域泛化性方面仍然落后于它们。具体来说，GPT-4V倾向于详细阐述解释和中间步骤，而Gemini则偏好输出直接简洁的答案。在流行的MME基准上的定量评估也证明了Gemini有潜力成为GPT-4V的强大挑战者。我们对Gemini的早期调查还观察到MLLMs的一些常见问题，表明距离人工通用智能仍有相当长的路要走。我们用于跟踪MLLM进展的项目已发布在https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models。

英文摘要：

The surge of interest towards Multi-modal Large Language Models (MLLMs), e.g., GPT-4V(ision) from OpenAI, has marked a significant trend in both academia and industry. They endow Large Language Models (LLMs) with powerful capabilities in visual understanding, enabling them to tackle diverse multi-modal tasks. Very recently, Google released Gemini, its newest and most capable MLLM built from the ground up for multi-modality. In light of the superior reasoning capabilities, can Gemini challenge GPT-4V's leading position in multi-modal learning? In this paper, we present a preliminary exploration of Gemini Pro's visual understanding proficiency, which comprehensively covers four domains: fundamental perception, advanced cognition, challenging vision tasks, and various expert capacities. We compare Gemini Pro with the state-of-the-art GPT-4V to evaluate its upper limits, along with the latest open-sourced MLLM, Sphinx, which reveals the gap between manual efforts and black-box systems. The qualitative samples indicate that, while GPT-4V and Gemini showcase different answering styles and preferences, they can exhibit comparable visual reasoning capabilities, and Sphinx still trails behind them concerning domain generalizability. Specifically, GPT-4V tends to elaborate detailed explanations and intermediate steps, and Gemini prefers to output a direct and concise answer. The quantitative evaluation on the popular MME benchmark also demonstrates the potential of Gemini to be a strong challenger to GPT-4V. Our early investigation of Gemini also observes some common issues of MLLMs, indicating that there still remains a considerable distance towards artificial general intelligence. Our project for tracking the progress of MLLM is released at https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。