The evolution from Large Language Models (LLMs) to Multimodal Large Language Models (MLLMs) has spurred research into extending In-Context Learning (ICL) to its multimodal counterpart. Existing such studies have primarily concentrated on image-to-text ICL. However, the Text-to-Image ICL (T2I-ICL), with its unique characteristics and potential applications, remains underexplored. To address this gap, we formally define the task of T2I-ICL and present CoBSAT, the first T2I-ICL benchmark dataset, encompassing ten tasks. Utilizing our dataset to benchmark six state-of-the-art MLLMs, we uncover considerable difficulties MLLMs encounter in solving T2I-ICL. We identify the primary challenges as the inherent complexity of multimodality and image generation, and show that strategies such as fine-tuning and Chain-of-Thought prompting help to mitigate these difficulties, leading to notable improvements in performance. Our code and dataset are available at https://github.com/UW-Madison-Lee-Lab/CoBSAT.
Can Multimodal Large Language Model Think Analogically?
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Shandong Institutes of Industrial Technology(山东产业技术研究院)
Traj-MLLM: Can Multimodal Large Language Models Reform Trajectory Data Mining?
University of Chinese Academy of Sciences(中国科学院大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Department of Computer Science, Aalborg University(奥胡斯大学计算机科学系);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院)
The Instinctive Bias: Spurious Images lead to Illusion in MLLMs
The Hong Kong Polytechnic University(香港理工大学);The Hong Kong University of Science and Technology(香港科技大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);NVIDIA(英伟达)
VFA: Empowering Multilingual MLLMs via Vision-Free Adaptation
Southern University of Science and Technology(南方科技大学);Microsoft Research Asia(微软亚洲研究院);Shanghai University of Finance and Economics(上海财经大学);Peking University(北京大学)
Unleashing Spatial Reasoning in Multimodal Large Language Models via Textual Representation Guided Reasoning
College of AI, Tsinghua University, Beijing, China(清华大学人工智能学院,北京,中国);Shanghai Artificial Intelligence Laboratory, Shanghai, China(上海人工智能实验室,上海,中国);The University of Tokyo, Tokyo, Japan(东京大学,东京,日本)
An Evaluation of Interleaved Instruction Tuning on Semantic Reasoning Performance in an Audio MLLM
The Graduate Center, CUNY(CUNY研究生中心);Brooklyn College, CUNY(CUNY布鲁克林学院);Borough of Manhattan Community College, CUNY(CUNY曼哈顿社区学院);The City College of New York, CUNY(CUNY纽约城市学院)
Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMs
School of Computer Science and Statistics, Trinity College Dublin(都柏林圣三一学院计算机科学与统计学院);Department of Mathematics and Computer Science, Eindhoven University of Technology(埃因霍温理工大学数学与计算机科学系)
Human-centered Interactive Learning via MLLMs for Text-to-Image Person Re-identification
College of Computer Science, Sichuan University(四川大学计算机学院);National Key Laboratory of Fundamental Algorithms and Models for Engineering Simulation, Sichuan University(四川大学国家工程仿真基础算法与模型重点实验室);Sichuan National Innovation New Vision UHD Video Technology Co., Ltd(四川国家创新新视觉超高清视频技术有限公司);Tianfu Jincheng Laboratory(天府锦城实验室)
MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steering
University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学);The Hong Kong Polytechnic University(香港理工大学);Huawei(华为)