How Well Does GPT-4V(ision) Adapt to Distribution Shifts? A Preliminary Investigation
Microsoft Research Asia(微软亚洲研究院);Westlake University(西湖大学);School of Software, Shandong University(山东大学软件学院);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Australian National University(澳大利亚国立大学);The University of New South Wales(新南威尔士大学);Data61, CSIRO(联邦科学与工业研究组织Data61);Macquarie University(麦考瑞大学);The University of Sydney(悉尼大学);Carnegie Mellon University(卡内基梅隆大学)
Zhongyi Han, Guanglin Zhou, Rundong He, Jindong Wang, Tailin Wu, Yilong Yin, Salman Khan, Lina Yao, Tongliang Liu, Kun Zhang
备注:
added the investigation of Gemini. 66 pages, 41 figures
In machine learning, generalization against distribution shifts -- where deployment conditions diverge from the training scenarios -- is crucial, particularly in fields like climate modeling, biomedicine, and autonomous driving. The emergence of foundation models, distinguished by their extensive pretraining and task versatility, has led to an increased interest in their adaptability to distribution shifts. GPT-4V(ision) acts as the most advanced publicly accessible multimodal foundation model, with extensive applications across various domains, including anomaly detection, video understanding, image generation, and medical diagnosis. However, its robustness against data distributions remains largely underexplored. Addressing this gap, this study rigorously evaluates GPT-4V's adaptability and generalization capabilities in dynamic environments, benchmarking against prominent models like CLIP, LLaVA, and Gemini. We delve into GPT-4V's zero-shot generalization across 13 diverse datasets spanning natural, medical, and molecular domains. We further investigate its adaptability to controlled data perturbations and examine the efficacy of in-context learning as a tool to enhance its adaptation. Our findings delineate GPT-4V's capability boundaries in distribution shifts, shedding light on its strengths and limitations across various scenarios. Importantly, this investigation contributes to our understanding of how AI foundation models generalize to distribution shifts, offering pivotal insights into their adaptability and robustness. The code is publicly available at https://github.com/jameszhou-gl/gpt-4v-distribution-shift.
Preliminary Investigations of a Multi-Faceted Robust and Synergistic Approach in Semiconductor Electron Micrograph Analysis: Integrating Vision Transformers with Large Language and Multimodal Models
GPT as Psychologist? Preliminary Evaluations for GPT-4V on Visual Affective Computing
The Hong Kong University of Science & Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science & Technology(香港科技大学);Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院);Zhejiang University(浙江大学);Great Bay University(大湾区大学);Hangzhou Research Institute, Beihang University(北京航空航天大学杭州研究院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)
Generating Findings for Jaw Cysts in Dental Panoramic Radiographs Using a GPT-Based VLM: A Preliminary Study on Building a Two-Stage Self-Correction Loop with Structured Output (SLSO) Framework
Gifu University(岐阜大学);EyeTech Co., Ltd(EyeTech有限公司);Asahi University(朝日大学);Shiga University(滋贺大学);The University of Osaka Dental Hospital(大阪大学齿科医院)
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Microsoft Corporation(微软公司);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
WizardMath 通过 RLEIF 方法在不依赖外部 Python 工具的情况下提升大语言模型的数学推理能力,在 GSM8k 和 MATH 基准上超越多个顶级开源及商用模型。
Can GPT-4 Help Detect Quit Vaping Intentions? An Exploration of Automatic Data Annotation Approach
HI3 Tech Lab, University of South Carolina(南卡罗来纳大学HI3技术实验室);School of Medicine, Washington University(华盛顿大学医学院);University of Texas Health Science Center(德克萨斯大学健康科学中心)
Dr. GPT Will See You Now, but Should It? Exploring the Benefits and Harms of Large Language Models in Medical Diagnosis using Crowdsourced Clinical Cases
Identifying and Decomposing Compound Ingredients in Meal Plans Using Large Language Models
Faculty of Health Sciences, University of Maribor(马里博尔大学健康科学学院);Faculty of Electrical Engineering and Computer Science, University of Maribor(马里博尔大学电气工程与计算机科学学院);The Nu B.V.(The Nu有限公司);Pain Research & Intervention Center of Excellence (PRICE), University of Florida(佛罗里达大学疼痛研究与干预卓越中心(PRICE));Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida(佛罗里达大学医学院健康结果与生物医学信息学系);Usher Institute, University of Edinburgh(爱丁堡大学厄舍研究所)
Research Entity Extraction and Topic Detection from UKRI Grant Proposals
EPCC, University of Edinburgh, UK(埃克塞特大学埃平中心,英国);Knowledge Media Institute, The Open University, UK(开放大学知识媒体研究所,英国);Institute for Technology and Humanity, University of Cambridge, UK(剑桥大学技术与人类研究所,英国);Centre for Higher Education Transformations, School of Education, University of Bristol, UK(布里斯托大学教育学院高等教育转型中心,英国);University of Oxford, UK(牛津大学,英国)