Information retrieval is indispensable for today's Internet applications, yet traditional semantic matching techniques often fall short in capturing the fine-grained cross-modal interactions required for complex queries. Although late-fusion two-tower architectures attempt to bridge this gap by independently encoding visual and textual data before merging them at a high level, they frequently overlook the subtle interplay essential for comprehensive understanding. In this work, we rigorously assess these limitations and introduce a unified retrieval framework that fuses visual and textual cues from the ground up, enabling early cross-modal interactions for enhancing context interpretation. Through a two-stage training process--comprising post-training adaptation followed by instruction tuning--we adapt MLLMs as retrievers using a simple one-tower architecture. Our approach outperforms conventional methods across diverse retrieval scenarios, particularly when processing complex multi-modal inputs. Notably, the joint fusion encoder yields greater improvements on tasks that require modality fusion compared to those that do not, underscoring the transformative potential of early integration strategies and pointing toward a promising direction for contextually aware and effective information retrieval.
We present MMFT-BERT(MultiModal Fusion Transformer with BERT encodings), to solve Visual Question Answering (VQA) ensuring individual and combined processing of multiple input modalities. Our approach benefits from processing multimodal data (video and text) adopting the BERT encodings individually and using a novel transformer-based fusion method to fuse them together. Our method decomposes the different sources of modalities, into different BERT instances with similar architectures, but variable weights. This achieves SOTA results on the TVQA dataset. Additionally, we provide TVQA-Visual, an…
NKIARI & SLAI, Shenzhen Futian and VCIP, Nankai University, Tianjin, China（NKIARI与SLAI，深圳福田及VCIP，南开大学，天津，中国）；Australian National University (ANU), Canberra, Australia（澳大利亚国立大学（ANU），堪培拉，澳大利亚）；King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia（国王阿卜杜勒·阿齐兹大学（KAUST），图瓦尔，沙特阿拉伯）；Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore（高性能计算研究所（IHPC），科学、技术和研究局（A*STAR），新加坡）
BTCChat: Advancing Remote Sensing Bi-temporal Change Captioning with Multimodal Large Language Model
State Key Laboratory of Networking and Switching Technology（网络与交换技术国家重点实验室）；Beijing University of Posts and Telecommunications（北京邮电大学）；Aerospace Information Research Institute（航天信息研究所）；Chinese Academy of Sciences（中国科学院）；School of Geographical Sciences（地理科学学院）；Hunan Normal University（湖南师范大学）
Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learning
Zhejiang University（浙江大学）；Fudan University（复旦大学）；Soochow University（苏州大学）；Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）
Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward
University of Science and Technology of China（中国科学技术大学）；State Key Laboratory of Cognitive Intelligence（认知智能国家重点实验室）；iFLYTEK AI Research（iFLYTEK人工智能研究院）；Hong Kong University of Science and Technology（香港科学与技术大学）
Understand, Think, and Answer: Advancing Visual Reasoning with Large Multimodal Models
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Peng Cheng Laboratory, Shenzhen, China（鹏城实验室）；Wuhan AI Research, Wuhan, China（武汉人工智能研究所）
University of Science and Technology of China（中国科学技术大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Inspired by the fact that different modalities in videos carry complementary information, we propose a Multimodal Semantic Attention Network(MSAN), which is a new encoder-decoder framework incorporating multimodal semantic attributes for video captioning. In the encoding phase, we detect and generate multimodal semantic attributes by formulating it as a multi-label classification problem. Moreover, we add auxiliary classification loss to our model that can obtain more effective visual features and high-level multimodal semantic attribute distributions for sufficient video encoding. In the deco…
Advancing Object Detection in Transportation with Multimodal Large Language Models (MLLMs): A Comprehensive Review and Empirical Testing
Arab American University（阿拉伯美国大学）；Columbia University（哥伦比亚大学）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）；Al-Ahliyya Amman University（阿赫利耶阿曼大学）；Queensland University of Technology（昆士兰科技大学）
Beyond Sequential Distance: Inter-Modal Distance Invariant Position Encoding
MAIS, Institute of Automation, Chinese Academy of Sciences, China（MAIS，自动化研究所，中国科学院，中国）；University of Chinese Academy of Sciences, China（中国科学院大学，中国）；Tencent Hunyuan Team, China（腾讯文言团队，中国）
IAD-GPT: Advancing Visual Knowledge in Multimodal Large Language Model for Industrial Anomaly Detection
School of Computer Science & Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；School of Computing and Information Technology, Great Bay University（大亚湾大学计算机与信息科技学院）；College of Computer Science, Nankai University（南开大学计算机学院）；School of Artificial Intelligence, Shenzhen University（深圳大学人工智能学院）；Guangdong Provincial Key Laboratory of Intelligent Information Processing, Shenzhen University（广东省智能信息处理重点实验室）；National Engineering Laboratory of Big Data System Computing Technology, Shenzhen University（大数据系统计算技术国家工程实验室）
Surgery-R1: Advancing Surgical-VQLA with Reasoning Multimodal Large Language Model via Reinforcement Learning
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong University of Science and Technology（香港科技大学）；Department of Thoracic Surgery, the Seventh Affiliated Hospital, Sun Yat-sen University（中山大学第七附属医院胸外科部门）；Bioengineering/Imperial-X, Imperial College London（生物工程/Imperial-X，帝国理工学院伦敦分校）；ROAS Thrust, Hong Kong University of Science and Technology (Guangzhou)（ROAS项目，香港科技大学（广州））；Department of Electronic and Computer Engineering, Hong Kong SAR（香港特别行政区电子与计算机工程系）
Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referring
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Peng Cheng Laboratory, Shenzhen, China（鹏城实验室）；Wuhan AI Research, Wuhan, China（武汉人工智能研究所）
提出统一高分辨率通用模型 Griffon v2，通过轻量下采样投影器和即插即用视觉分词器实现视觉-语言共指代，在多项视觉感知任务上达到 SOTA 并超越专家模型。
Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens
AIML, University of Adelaide（AIML，阿德莱德大学）；Monash University（墨尔本大学）；Australian National University（澳大利亚国立大学）；Tiktok, Australia（Tiktok，澳大利亚）；Zhejiang University（浙江大学）
LMM-VQA: Advancing Video Quality Assessment with Large Multimodal Models
Institute of Image Communication and Information Processing, Shanghai Jiao Tong University（上海交通大学图像通信与信息处理研究所）；MoE Key Laboratory of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院（教育部人工智能重点实验室））；Huawei Technologies（华为技术有限公司）
ViTaPEs: Visuotactile Position Encodings for Cross-Modal Alignment in Multimodal Transformers
Chair of Cyber-Physical Systems（感知系统教授席位）；Institute of Machine Learning and Neural Computation（机器学习与神经计算研究所）；Technical University of Leoben（莱比锡技术大学）；Graz University of Technology（格拉茨技术大学）
Advancing Grounded Multimodal Named Entity Recognition via LLM-Based Reformulation and Box-Based Segmentation
College of Intelligence and Computing, Tianjin University（智能与计算学院，天津大学）；NJUST（南京理工大学）；College of Mathematics, Taiyuan University of Technology（数学学院，太原科技大学）；Tianjin University of Science and Technology（天津科技大学）
iGVLM: Dynamic Instruction-Guided Vision Encoding for Question-Aware Multimodal Understanding
School of Computer Science（计算机科学学院）；Technology, Huazhong University of Science（科技，华中科技大学）；Li Auto Inc.（Li汽车公司）；Institute of AI for Industries, Chinese Academy of Sciences（产业人工智能研究所，中国科学院）
C^2ROPE: Causal Continuous Rotary Positional Encoding for 3D Large Multimodal-Models Reasoning
State Key Laboratory of Internet of Things for Smart City, University of Macau（物联网智能城市国家重点实验室，澳门大学）；Department of Automation, Shanghai Jiaotong University（上海交通大学自动化系）；Institute of Advanced Technology, University of Science and Technology of China（中国科学技术大学先进技术研究院）；School of Computer Science and Technology, USTC（中国科学技术大学计算机科学与技术学院）；School of Artificial Intelligence and Data Science, USTC（中国科学技术大学人工智能与数据科学学院）
From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
School of Computer Science, Peking University（北京大学计算机学院）；The Chinese University of Hong Kong（香港中文大学）；University of California, San Diego（加利福尼亚大学圣迭戈分校）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
MM-ISTS: Cooperating Irregularly Sampled Time Series Forecasting with Multimodal Vision-Text LLMs
East China Normal University Shanghai China；Centre for Artificial Intelligence；Robotics, Hong Kong Institute of Science \& Innovation, Chinese Academy of Sciences Hong Kong China；Department of Computing, The Hong Kong Polytechnic University Hong Kong China；East China Normal University；Robotics, Hong Kong Institute of Science \& Innovation, Chinese Academy of Sciences；Department of Computing, The Hong Kong Polytechnic University
Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers
Archimedes, Athena Research Center（阿基米德研究中心）；National Technical University of Athens（希腊国家技术大学）；University of Crete（克里特大学）；IACM-Forth（第四研究机构（IACM-Forth））
Most approaches for semantic segmentation use only information from color cameras to parse the scenes, yet recent advancements show that using depth data allows to further improve performances. In this work, we focus on transformer-based deep learning architectures, that have achieved state-of-the-art performances on the segmentation task, and we propose to employ depth information by embedding it in the positional encoding. Effectively, we extend the network to multimodal data without adding any parameters and in a natural way that makes use of the strength of transformers' self-attention mod…
HyPCA-Net: Advancing Multimodal Fusion in Medical Image Analysis
Indian Institute of Technology Ropar（印度理工学院罗帕尔分校）；RoentGen Health（RoentGen健康公司）；Lulea University of Technology（卢勒奥大学）；QUT（昆士兰科技大学）；RMIT University（皇家墨尔本理工大学）；Curtin University（Curtin大学）；Deakin University（德肯大学）
