arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.07424cs.LGcs.AIcs.CV
收藏

GPT-4V(ision) 对分布偏移的适应能力如何?一项初步研究

How Well Does GPT-4V(ision) Adapt to Distribution Shifts? A Preliminary Investigation

Microsoft Research Asia(微软亚洲研究院);Westlake University(西湖大学);School of Software, Shandong University(山东大学软件学院);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Australian National University(澳大利亚国立大学);The University of New South Wales(新南威尔士大学);Data61, CSIRO(联邦科学与工业研究组织Data61);Macquarie University(麦考瑞大学);The University of Sydney(悉尼大学);Carnegie Mellon University(卡内基梅隆大学)

Zhongyi Han, Guanglin Zhou, Rundong He, Jindong Wang, Tailin Wu, Yilong Yin, Salman Khan, Lina Yao, Tongliang Liu, Kun Zhang

AI总结:

本文通过13个跨领域数据集和受控扰动实验,系统评估了GPT-4V在分布偏移下的零样本泛化与上下文学习适应能力,并与CLIP、LLaVA、Gemini对比,揭示了其能力边界与鲁棒性。

摘要:

在机器学习中,对分布偏移的泛化能力——即部署条件与训练场景发生偏离的情况——至关重要,尤其是在气候建模、生物医学和自动驾驶等领域。基础模型的出现以其大规模预训练和任务通用性为特征,引发了人们对其适应分布偏移能力的日益关注。GPT-4V(ision) 是目前最先进的公开可访问多模态基础模型,在异常检测、视频理解、图像生成和医学诊断等众多领域有着广泛应用。然而,其对数据分布的鲁棒性在很大程度上仍未得到充分探索。为填补这一空白,本研究严格评估了 GPT-4V 在动态环境中的适应性和泛化能力,并以 CLIP、LLaVA 和 Gemini 等知名模型为基准进行对比。我们深入考察了 GPT-4V 在涵盖自然、医学和分子领域的 13 个多样化数据集上的零样本泛化表现。我们进一步研究了其对受控数据扰动的适应能力,并检验了上下文学习作为增强其适应性的工具的有效性。我们的研究结果勾勒出 GPT-4V 在分布偏移下的能力边界,揭示了其在不同场景中的优势与局限。重要的是,这项研究有助于我们理解 AI 基础模型如何泛化到分布偏移,为其适应性和鲁棒性提供了关键见解。代码已公开于 https://github.com/jameszhou-gl/gpt-4v-distribution-shift。

英文摘要:

In machine learning, generalization against distribution shifts -- where deployment conditions diverge from the training scenarios -- is crucial, particularly in fields like climate modeling, biomedicine, and autonomous driving. The emergence of foundation models, distinguished by their extensive pretraining and task versatility, has led to an increased interest in their adaptability to distribution shifts. GPT-4V(ision) acts as the most advanced publicly accessible multimodal foundation model, with extensive applications across various domains, including anomaly detection, video understanding, image generation, and medical diagnosis. However, its robustness against data distributions remains largely underexplored. Addressing this gap, this study rigorously evaluates GPT-4V's adaptability and generalization capabilities in dynamic environments, benchmarking against prominent models like CLIP, LLaVA, and Gemini. We delve into GPT-4V's zero-shot generalization across 13 diverse datasets spanning natural, medical, and molecular domains. We further investigate its adaptability to controlled data perturbations and examine the efficacy of in-context learning as a tool to enhance its adaptation. Our findings delineate GPT-4V's capability boundaries in distribution shifts, shedding light on its strengths and limitations across various scenarios. Importantly, this investigation contributes to our understanding of how AI foundation models generalize to distribution shifts, offering pivotal insights into their adaptability and robustness. The code is publicly available at https://github.com/jameszhou-gl/gpt-4v-distribution-shift.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑