arXiv 2401.150712024-01-29cs.CV

从 GPT-4 到 Gemini 及更远：通过四种模态评估 MLLMs 在泛化性、可信性和因果性方面的研究格局

From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities

National University of Singapore（新加坡国立大学）；Shanghai AI Laboratory（上海人工智能实验室）

Chaochao Lu, Chen Qian, Guodong Zheng, Hongxing Fan, Hongzhi Gao, Jie Zhang, Jing Shao, Jingyi Deng, Jinlan Fu, Kexin Huang, Kunchang Li, Lijun Li, Limin Wang, … 展开作者

Chaochao Lu, Chen Qian, Guodong Zheng, Hongxing Fan, Hongzhi Gao, Jie Zhang, Jing Shao, Jingyi Deng, Jinlan Fu, Kexin Huang, Kunchang Li, Lijun Li, Limin Wang, Lu Sheng, Meiqi Chen, Ming Zhang, Qibing Ren, Sirui Chen, Tao Gui, Wanli Ouyang, Yali Wang, Yan Teng, Yaru Wang, Yi Wang, Yinan He, Yingchun Wang, Yixu Wang, Yongting Zhang, Yu Qiao, Yujiong Shen, Yurong Mou, Yuxi Chen, Zaibin Zhang, Zhelun Shi, Zhenfei Yin, Zhipin Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文通过文本、代码、图像、视频四种模态，定性评估闭源与开源多模态大语言模型的泛化性、可信性和因果推理能力，得出14项实证发现，为理解模型能力边界、构建更可靠的下游多模态应用提供参考。

摘要：

多模态大语言模型（Multi-modal Large Language Models, MLLMs）在针对多模态内容生成合理响应方面已展现出令人瞩目的能力。然而，尽管最强大的 OpenAI GPT-4 和 Google Gemini 已经部署，当前基于 MLLM 的应用性能与广大公众的期望之间仍存在较大差距。本文通过对近期商用闭源与开源 MLLMs 在文本、代码、图像、视频四种模态下的泛化性、可信性和因果推理能力开展定性研究，致力于加深对这一差距的理解，最终目标是提升 MLLMs 的透明度。我们认为这些属性是定义 MLLMs 在支撑各类下游应用时可靠性的几个代表性因素。具体而言，我们评估了闭源的 GPT-4、Gemini 以及 6 个开源 LLM 和 MLLM。我们总共评估了 230 个人工设计的案例，定性结果随后被汇总为 12 项得分（即 4 种模态乘以 3 项属性）。我们总共得出了 14 项实证发现，这些发现有助于理解商用与开源 MLLMs 的能力与局限性，以推动构建更可靠的下游多模态应用。

英文摘要：

Multi-modal Large Language Models (MLLMs) have shown impressive abilities in generating reasonable responses with respect to multi-modal contents. However, there is still a wide gap between the performance of recent MLLM-based applications and the expectation of the broad public, even though the most powerful OpenAI's GPT-4 and Google's Gemini have been deployed. This paper strives to enhance understanding of the gap through the lens of a qualitative study on the generalizability, trustworthiness, and causal reasoning capabilities of recent proprietary and open-source MLLMs across four modalities: ie, text, code, image, and video, ultimately aiming to improve the transparency of MLLMs. We believe these properties are several representative factors that define the reliability of MLLMs, in supporting various downstream applications. To be specific, we evaluate the closed-source GPT-4 and Gemini and 6 open-source LLMs and MLLMs. Overall we evaluate 230 manually designed cases, where the qualitative results are then summarized into 12 scores (ie, 4 modalities times 3 properties). In total, we uncover 14 empirical findings that are useful to understand the capabilities and limitations of both proprietary and open-source MLLMs, towards more reliable downstream multi-modal applications.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。