Singapore Management University(新加坡管理大学);Southwest Jiaotong University(西南交通大学);Singapore University of Technology and Design(新加坡科技设计大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Lei Wang, Wanyu Xu, Zhiqiang Hu, Yihuai Lan, Shan Dong, Hao Wang, Roy Ka-Wei Lee, Ee-Peng Lim
This paper introduces a new in-context learning (ICL) mechanism called In-Image Learning (I$^2$L) that combines demonstration examples, visual cues, and chain-of-thought reasoning into an aggregated image to enhance the capabilities of Large Multimodal Models (e.g., GPT-4V) in multimodal reasoning tasks. Unlike previous approaches that rely on converting images to text or incorporating visual input into language models, I$^2$L consolidates all information into an aggregated image and leverages image processing, understanding, and reasoning abilities. This has several advantages: it reduces inaccurate textual descriptions of complex images, provides flexibility in positioning demonstration examples, and avoids multiple input images and lengthy prompts. We also introduce I$^2$L-Hybrid, a method that combines the strengths of I$^2$L with other ICL methods. Specifically, it uses an automatic strategy to select the most suitable method (I$^2$L or another certain ICL method) for a specific task instance. We conduct extensive experiments to assess the effectiveness of I$^2$L and I$^2$L-Hybrid on MathVista, which covers a variety of complex multimodal reasoning tasks. Additionally, we investigate the influence of image resolution, the number of demonstration examples in a single image, and the positions of these demonstrations in the aggregated image on the effectiveness of I$^2$L. Our code is publicly available at https://github.com/AGI-Edgerunners/IIL.
Beyond Demonstrations: Dynamic Vector Construction from Latent Representations
National Key Laboratory for Multimedia Information Processing, Peking University(国家多媒体信息处理重点实验室,北京大学);MOE Key Laboratory of Computational Linguistics, Peking University(教育部计算语言学重点实验室,北京大学);School of Software and Microelectronics, Peking University(软件与微电子学院,北京大学);School of Computer Science, Peking University(计算机科学学院,北京大学)
Representation-guided in-context learning for medical image interpretation with multimodal large language models
Schepens Eye Research Institute of Massachusetts Eye and Ear(麻省眼耳医院谢本斯眼科研究所);Harvard Medical School(哈佛医学院);Massachusetts Eye and Ear(麻省眼耳医院);National Taiwan University Hospital(台湾大学医院);Massachusetts Institute of Technology(麻省理工学院);Northwestern University(西北大学);Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University(哈佛大学肯普纳自然与人工智能研究所);Harvard University(哈佛大学);Harvard Data Science Initiative, Harvard University(哈佛大学数据科学倡议)
Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis
School of Computer Science & Engineering, Key Lab of New Generation Artificial Intelligence Technology & Its Interdisciplinary Applications (Ministry of Education), Southeast University, China(计算机科学与工程学院,新一代人工智能技术及跨学科应用重点实验室,东南大学,中国);Faculty of IT, Monash University, Australia(信息科技学院,墨尔本大学,澳大利亚);School of Computer Science & Engineering, Southeast University, China(计算机科学与工程学院,东南大学,中国)
Smart Eyes for Silent Threats: VLMs and In-Context Learning for THz Imaging
Data and Web Science Group, University of Mannheim(曼海姆大学数据与网络科学组);Max-Planck-Institute for Informatics, Saarland Informatics Campus(马克斯·普朗克信息研究所,萨尔兰州信息校区)
Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning
Hacettepe University(哈杰泰佩大学);Aselsan Research(阿塞尔桑研究院);Koç University(科克大学);Amsterdam UMC, University of Amsterdam(阿姆斯特丹大学医学中心,阿姆斯特丹大学);Amsterdam Public Health(阿姆斯特丹公共卫生研究院)
Efficient Universal Models for Medical Image Segmentation via Weakly Supervised In-Context Learning
Harbin Institute of Technology at Shenzhen(哈尔滨工业大学(深圳));Peng Cheng Laboratory(鹏城实验室);University Hospital Tübingen(图宾根大学医院);German Center for Mental Health(德国心理健康中心)
Pose-ICL: 3D-Aware In-Context Learning for Pose-Controllable Subject Customization
College of Electronic and Information Engineering, Tongji University, Shanghai, China(同济大学电子与信息工程学院);State Key Laboratory of Autonomous Intelligent Unmanned Systems, Frontiers Science Center for Intelligent Autonomous Systems, Ministry of Education, Shanghai, China(自主智能无人系统国家重点实验室,智能自主系统前沿科学中心,教育部,上海,中国)
Think Bright, Diffuse Nice: Enhancing T2I-ICL via Inductive-Bias Hint Instruction and Query Contrastive Decoding
Cao Tu Li (Guangzhou) Technology Co., Ltd(曹图利(广州)科技有限公司);South China University of Technology(华南理工大学);Washington State University(华盛顿州立大学);Hong Kong Baptist University(香港 Baptist 大学)
Towards Multimodal In-Context Learning for Vision & Language Models
IBM Research(IBM研究院);Weizmann Institute of Science(魏茨曼科学研究所);TU Graz(格拉茨工业大学);ELLIS Unit(ELLIS单元);Institute for Machine Learning, JKU Linz(林茨约翰内斯·开普勒大学机器学习研究所);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室)
Cycle Context Verification for In-Context Medical Image Segmentation
National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science;Engineering, Northwestern Polytechnical University, Xi’an 710072, China Institute of High Performance Computing, Agency for Science, Technology;Research, Singapore 138632, Singapore Ningbo Institute of Northwestern Polytechnical University, Ningbo 315048, China Research \& Development Institute of Northwestern Polytechnical University in Shenzhen, Shenzhen 518057, China
Interventional Contrastive Learning with Meta Semantic Regularizer
Institute of Software Chinese Academy of Sciences(中国科学院软件研究所);University of Chinese Academy of Sciences(中国科学院大学);Southern Marine Science and Engineering Guangdong Laboratory (Guangzhou)(南方海洋科学与工程广东省实验室(广州));Renmin University of China(中国人民大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学)
In-Context Learning for Wound Classification with Small Multimodal Language Models
Jönköping University(延雪平大学);Halmstad University(哈尔姆斯塔德大学);Mölnlycke Health Care(墨尼克医疗保健公司);Centre for Reliable Machine Learning, Royal Holloway, University of London(伦敦大学皇家霍洛威学院可靠机器学习中心)
Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?
LMU Munich(慕尼黑大学);Siemens AG(西门子公司);Technical University of Munich(慕尼黑工业大学);University of Oxford(牛津大学);University of California, Santa Barbara(加州大学圣巴巴拉分校)
Towards Robust In-Context Learning for Medical Image Segmentation via Data Synthesis
Harbin Institute of Technology at Shenzhen(哈尔滨工业大学(深圳));Peng Cheng Laboratory(鹏城实验室);University Hospital Tübingen(图宾根大学医院);German Center for Mental Health(德国心理健康中心)
Uncovering local aggregated air quality index with smartphone captured images leveraging efficient deep convolutional neural network
University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);BRAC University(BRAC大学);Dublin City University(都柏林城市大学);Bangladesh University of Engineering and Technology(孟加拉国工程技术大学)