Leveraging Chat-Based Large Vision Language Models for Multimodal Out-Of-Context Detection
The Graduate University for Advanced Studies, SOKENDAI（总合研究大学院大学（ Sokendai））；National Institute of Informatics（情报通信研究机构信息学研究所）；The University of Tokyo（东京大学）
Fatma Shalabi, Hichem Felouat, Huy H. Nguyen, Isao Echizen
Out-of-context (OOC) detection is a challenging task involving identifying images and texts that are irrelevant to the context in which they are presented. Large vision-language models (LVLMs) are effective at various tasks, including image classification and text generation. However, the extent of their proficiency in multimodal OOC detection tasks is unclear. In this paper, we investigate the ability of LVLMs to detect multimodal OOC and show that these models cannot achieve high accuracy on OOC detection tasks without fine-tuning. However, we demonstrate that fine-tuning LVLMs on multimodal OOC datasets can further improve their OOC detection accuracy. To evaluate the performance of LVLMs on OOC detection tasks, we fine-tune MiniGPT-4 on the NewsCLIPpings dataset, a large dataset of multimodal OOC. Our results show that fine-tuning MiniGPT-4 on the NewsCLIPpings dataset significantly improves the OOC detection accuracy in this dataset. This suggests that fine-tuning can significantly improve the performance of LVLMs on OOC detection tasks.
ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Bytedance Inc.（字节跳动公司）
Leveraging ChatGPT's Multimodal Vision Capabilities to Rank Satellite Images by Poverty Level: Advancing Tools for Social Science Research
Center for Applied Intelligent Systems Research (CAISR), Halmstad University, Sweden（应用智能系统研究中心（CAISR），哈马斯特德大学，瑞典）；Department of Human Geography, Lund University, Sweden（人类地理系，吕勒奥大学，瑞典）；Department of Earth and Environmental Sciences, Lund University, Sweden（地球与环境科学系，吕勒奥大学，瑞典）
Accountable Textual-Visual Chat Learns to Reject Human Instructions in Image Re-creation
The Chinese University of Hong Kong（香港中文大学）；Centre for Perceptual and Interactive Intelligence（感知与交互智能中心）；Huazhong University of Science and Technology（华中科技大学）
ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model
Midea Group（美的集团）；East China Normal University（华东师范大学）；Shanghai University（上海大学）；Beijing Innovation Center of Humanoid Robotics（北京人形机器人创新中心）；Tsinghua University（清华大学）
Can ChatGPT Detect DeepFakes? A Study of Using Multimodal Large Language Models for Media Forensics
University at Buffalo, State University of New York（纽约州立大学布法罗分校）；Williamsville East High School（威廉斯维尔东高中）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University at Albany, State University of New York（纽约州立大学奥尔巴尼分校）
本文研究多模态 LLM 能否用于 DeepFake 检测，通过定性与定量实验及提示工程证明其可识别 AI 生成图像，并分析局限与改进方向。
ChatSR: Multimodal Large Language Models for Scientific Formula Discovery
AnnLab, Institute of Semiconductors, Chinese Academy of Sciences, Beijing, China（安 lab，半导体研究所，中国科学院，北京，中国）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing, China（电子、电气与通信工程学院，中国科学院大学，北京，中国）；Zhongguancun Academy, Beijing, China（中关村学院，北京，中国）；School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing 101408, China（先进交叉科学学院，中国科学院大学，北京101408，中国）；College of Materials Science and Opto-Electronic Technology, University of Chinese Academy of Sciences, Beijing, 100049, China（材料科学与光电技术学院，中国科学院大学，北京100049，中国）；School of Integrated Circuits, University of Chinese Academy of Sciences, Beijing 100049, China（集成电路学院，中国科学院大学，北京100049，中国）
A Multimodal Manufacturing Safety Chatbot: Knowledge Base Design, Benchmark Development, and Evaluation of Multiple RAG Approaches
Farmer School of Business, Miami University（Miami大学农业商学院）；Department of Computer Science and Software Engineering, Miami University（Miami大学计算机科学与软件工程系）；Department of Engineering Technology, Miami University（Miami大学工程技术系）；Department of Mechanical and Manufacturing Engineering, Miami University（Miami大学机械与制造工程系）；Department of Industrial and Systems Engineering, University at Buffalo（University at Buffalo工业与系统工程系）
Evaluation of Medical Vision Language Models HuluMed and MedGemma, and general purpose chatbots Gemma 3, ChatGPT Plus, and Claude Pro on real previously unseen wound images
Department of Computer Science, New Jersey Institute of Technology（新泽西理工学院计算机科学系）；Vascular and Endovascular Surgery, Robert Wood Johnson Hospital（罗伯特·伍德·约翰逊医院血管外科）；Department of Data Science, New Jersey Institute of Technology（新泽西理工学院数据科学系）
本研究评估了六种视觉语言模型在慢性伤口分析任务上的表现，发现通用模型 ChatGPT 和 Claude 显著优于医学专用模型，表明广泛的多模态推理能力比领域知识更重要。
AugGPT: Leveraging ChatGPT for Text Data Augmentation
University of Georgia（佐治亚大学）；South China University of Technology（华南理工大学）；Lehigh University（利哈伊大学）；Carnegie Mellon University（卡内基梅隆大学）；Mayo Clinic（梅奥诊所）；University of Virginia（弗吉尼亚大学）；The University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）
Leveraging Taxonomy and LLMs for Improved Multimodal Hierarchical Classification
University of New South Wales（新南威尔士大学）；Deakin University（迪肯大学）；University of Southampton（南安普顿大学）；Macquarie University（麦考瑞大学）；Technology Innovation Institute（技术创新研究院）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）
Reasoning Guided Embeddings: Leveraging MLLM Reasoning for Improved Multimodal Retrieval
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；Sensetime Research（商汤科技研究院）；Beijing Institute of Technology（北京理工大学）；Shanghai AI Lab（上海人工智能实验室）
Leveraging ChatGPT and Other NLP Methods for Identifying Risk and Protective Behaviors in MSM: Social Media and Dating apps Text Analysis
1 Department of Computer Science, Henry Samueli School of Engineering, University of California, Los Angeles, Los Angeles, CA；2 Department of Social Welfare, Luskin School of Public Affairs, University of California, Los Angeles, Los Angeles, CA；3 Department of Counseling \& Educational Psychology, New Mexico State University, Las Cruces, NM
Chat-TS: Enhancing Multi-Modal Reasoning Over Time-Series and Natural Language Data
Electrical and Computer Engineering, Queen’s University（皇后大学电气与计算机工程学院）；Mechanical and Materials Engineering, Queen’s University（皇后大学机械与材料工程学院）；Ingenuity Labs Research Institute, Queen’s University（皇后大学创新实验室研究 institute）
ChatTracker: Enhancing Visual Tracking Performance via Chatting with Multimodal Large Language Model
School of Computer Science, Fudan University（复旦大学计算机科学学院）；Meituan Inc（美团公司）；Shanghai Frontiers Science Center of Molecule Intelligent Syntheses（上海分子智能合成前沿科学中心）
Percept, Chat, and then Adapt: Multimodal Knowledge Transfer of Foundation Models for Open-World Video Recognition
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；the School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Shanghai AI Laboratory（上海人工智能实验室）
