The increase in parameter size of multimodal large language models (MLLMs) introduces significant capabilities, particularly in-context learning, where MLLMs enhance task performance without updating pre-trained parameters. This effectiveness, however, hinges on the appropriate selection of in-context examples, a process that is currently biased towards visual data, overlooking textual information. Furthermore, the area of supervised retrievers for MLLMs, crucial for optimal in-context example selection, continues to be uninvestigated. Our study offers an in-depth evaluation of the impact of textual information on the unsupervised selection of in-context examples in multimodal contexts, uncovering a notable sensitivity of retriever performance to the employed modalities. Responding to this, we introduce a novel supervised MLLM-retriever MSIER that employs a neural network to select examples that enhance multimodal in-context learning efficiency. This approach is validated through extensive testing across three distinct tasks, demonstrating the method's effectiveness. Additionally, we investigate the influence of modalities on our supervised retrieval method's training and pinpoint factors contributing to our model's success. This exploration paves the way for future advancements, highlighting the potential for refined in-context learning in MLLMs through the strategic use of multimodal data.
VTPerception-R1: Enhancing Multimodal Reasoning via Explicit Visual and Textual Perceptual Grounding
Fudan University（复旦大学）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Hong Kong（香港大学）；Shenzhen University（深圳大学）；University of Science and Technology of China（中国科学技术大学）
Enhancing textual textbook question answering with large language models and retrieval augmented generation
Faculty of Computing and Information Technology, King Abdulaziz University（阿卜杜勒阿齐兹国王大学计算与信息技术学院）；College of Computer and Information Science, Imam Mohammad Ibn Saud Islamic University (IMSIU)（伊玛目穆罕默德·本·沙特伊斯兰大学计算机与信息科学学院）；School of Computing, Macquarie University（麦考瑞大学计算学院）
MARA: A Multimodal Adaptive Retrieval-Augmented Framework for Document Question Answering
Key Laboratory of Target Cognition and Application Technology (TCAT), AIRI, CAS（目标认知与应用技术重点实验室（TCAT），空气动力研究所，中国科学院）；School of Electronic, Electrical and Communication Engineering, UCAS（电子、电气与通信工程学院，中国科学院大学）；Aerospace Information Research Institute, Chinese Academy of Sciences（航天信息研究所，中国科学院）；Baidu Inc.（百度公司）
ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data
Beijing University of Posts and Telecommunications（北京邮电大学）；Pengcheng Laboratory（鹏城实验室）；China Unicom Network Communications Corporation Limited（中国联合网络通信股份有限公司）
V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrieval
Tsinghua University（清华大学）；University of Central Florida（中央佛罗里达大学）；Fudan University（复旦大学）；The Australian National University（澳大利亚国立大学）；The University of Hong Kong（香港大学）；Pengcheng Laboratory（鹏城实验室）；Central South University（中南大学）
ChemVTS-Bench: Evaluating Visual-Textual-Symbolic Reasoning of Multimodal Large Language Models in Chemistry
Renmin University of China（中国人民大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；South China University of Technology（华南理工大学）；Gaotu Techedu Inc（高图科技公司）
Hidden in the Metadata: Stealth Poisoning Attacks on Multimodal Retrieval-Augmented Generation
The City University of New York, CSI, Staten Island, NY 10314, USA（纽约城市大学，CSI，史泰登岛分校）；The City University of New York, Graduate Center, New York, NY 10016, USA（纽约城市大学研究生中心）
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）；School of Computer Science, University of Sydney（悉尼大学计算机学院）；Alibaba Group（阿里巴巴集团）
Unveil: Unified Visual-Textual Integration and Distillation for Multi-modal Document Retrieval
State Key Laboratory of General Artificial Intelligence, Peking University（北京理工大学通用人工智能国家重点实验室）；School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航空航天信息研究所）；Key Laboratory of Target Cognition and Application Technology（目标认知与应用技术重点实验室）；Beijing Institute of Technology（北京理工大学）；Ucap Cloud（Ucap云）
