Medical large vision-language models (Med-LVLMs) have recently achieved remarkable progress in vision-language comprehension and medical image segmentation. However, existing models still struggle to unify these two capabilities, which is essential for achieving clinically reasoning that connects visual findings with semantic interpretation. We present MedSIGHT, a unified framework that equips Med-LVLMs with structured, pixel-level understanding for grounded visual comprehension. MedSIGHT introduces a novel Region Perceiver module that produces region-centric tokens, encoding spatial information directly into representation space of the language model. We further propose a medical region codebook into the LLM vocabulary, allowing the model to generate discrete region codes as symbolic representations of anatomical and pathological regions. These codes are decoded through the Region Perceiver to reconstruct segmentation mask, achieving end-to-end spatial grounding. Lastly, MedSIGHT combines Region Perceiver, Codebook and LLM using our proposed progressive training strategy to gradually aligns these modules stably. Trained on only 72K multimodal instruction pairs, MedSIGHT achieves state-of-the-art performance across diverse imaging modalities on both medical comprehension and segmentation tasks.
EyePCR: A Comprehensive Benchmark for Fine-Grained Perception, Knowledge Comprehension and Clinical Reasoning in Ophthalmic Surgery
School of Computer Science & Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；School of Computer Science, University of Nottingham Ningbo China（诺丁汉大学宁波校区计算机科学学院）；Wenzhou Medical University（温州医科大学）；School of Computer Science, University of Nottingham（诺丁汉大学计算机科学学院）
LongEgoRefer: A Benchmark for Long-Form Egocentric Video Referring Expression Comprehension
Woven by Toyota, Japan（丰田公司，日本）；The University of Tokyo, Japan（东京大学，日本）；National Institute of Informatics, Japan（日本信息机构国家研究所）；Institute of Science Tokyo, Japan（东京科学研究所，日本）；NII LLMC, Japan（日本信息机构LLMC）；Kyoto University, Japan（京都大学，日本）
GRACE: Boosting Video MLLMs with Grounded Action-Centric Evidence for Viewer Sentiment Prediction
Shanghai Jiao Tong University（上海交通大学）；Hangzhou Dianzi University（杭州电子科技大学）；The 52nd Research Institute of China Electronics Technology Group Corporation（中国电子科技集团公司第五十二研究所）；Hangzhou Bywin Technology Co., Ltd.（杭州百威科技有限公司）；Zhejiang Dahua Technology Co., Ltd.（浙江大华技术股份有限公司）；School of Information Science and Engineering, Shandong University（山东大学信息科学与工程学院）；Haihe Laboratory of Information Technology Application Innovation（海河信息技术应用创新实验室）
Uncovering Hidden Connections: Iterative Search and Reasoning for Video-grounded Dialog
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）；Pengcheng Laboratory（鹏城实验室）；School of Computer Science and Technology, Shandong Jianzhu University（山东建筑大学计算机科学与技术学院）
CodePercept: Code-Grounded Visual STEM Perception for MLLMs
MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University（人工智能大模型重点实验室，人工智能学院，计算机科学学院，上海交通大学）；Qwen Team（通义实验室）；Beijing Institute of Technology（北京理工大学）；Tsinghua University（清华大学）；Zhejiang University（浙江大学）
A Pathway to General-Purpose Scientific AI: Multimodal Comprehension of Scientific Images
TIB Leibniz Information Centre for Science and Technology（莱布尼茨科学与技术信息中心（TIB））；Eindhoven University of Technology（埃因霍温理工大学）；Freie Universität Berlin（柏林自由大学）；International Center for Chemical and Biological Sciences, University of Karachi（卡拉奇大学国际化学与生物科学中心）；National University of Sciences and Technology（国家科技大学）；University of Warwick（华威大学）；PSG College of Technology（PSG理工学院）；Aalto University（阿尔托大学）
NavBench: Probing Multimodal Large Language Models for Embodied Navigation
The University of Adelaide（阿德莱德大学）；The University of Queensland（昆士兰大学）；CSIRO Data61（澳大利亚联邦科学与工业研究组织数据61研究中心）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Tongji University（同济大学）
LLMs as Bridges: Reformulating Grounded Multimodal Named Entity Recognition
School of New Media and Communication, Tianjin University（天津大学新媒体与传播学院）；College of Mathematics, Taiyuan University of Technology（太原理工大学数学学院）；Tianjin University of Science and Technology（天津科技大学）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；University of Copenhagen（哥本哈根大学）
Stanford University（斯坦福大学）；Massachusetts Institute of Technology（麻省理工学院）
Grounding object properties and relations in 3D scenes is a prerequisite for a wide range of artificial intelligence tasks, such as visually grounded dialogues and embodied manipulation. However, the variability of the 3D domain induces two fundamental challenges: 1) the expense of labeling and 2) the complexity of 3D grounded language. Hence, essential desiderata for models are to be data-efficient, generalize to different data distributions and tasks with unseen semantic forms, as well as ground complex language semantics (e.g., view-point anchoring and multi-object reference). To address th…
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Innovation Center of Tsinghua University（清华大学创新中心）；National University of Singapore（新加坡国立大学）
Vision-language pre-training (VLP) has shown impressive performance on a wide range of cross-modal tasks, where VLP models without reliance on object detectors are becoming the mainstream due to their superior computation efficiency and competitive performance. However, the removal of object detectors also deprives the capability of VLP models in explicit object modeling, which is essential to various position-sensitive vision-language (VL) tasks, such as referring expression comprehension and visual commonsense reasoning. To address the challenge, we introduce PEVL that enhances the pre-train…
Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understanding
CUHK MMLab（香港中文大学MML实验室）；vivo AI Lab（vivo人工智能实验室）；CPII under InnoHK（创新工场下的CPII）；Shanghai AI Lab & Shenzhen Institute of Advanced Technology, CAS（上海人工智能实验室及深圳先进技术研究所，中国科学院）
Pengcheng Laboratory（鹏城实验室）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
We introduce a new multi-modal task for computer systems, posed as a combined vision-language comprehension challenge: identifying the most suitable text describing a scene, given several similar options. Accomplishing the task entails demonstrating comprehension beyond just recognizing "keywords" (or key-phrases) and their corresponding visual concepts. Instead, it requires an alignment between the representations of the two modalities that achieves a visually-grounded "understanding" of various linguistic elements and their dependencies. This new task also admits an easy-to-compute and well-…
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
International Digital Economy Academy (IDEA)（国际数字经济学院（IDEA））；Institute for AI, Tsinghua-Bosch Joint Center for ML, Tsinghua University（清华大学人工智能研究院、清华-博世联合机器学习中心）；The Hong Kong University of Science and Technology（香港科技大学）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Microsoft Research（微软研究院）；South China University of Technology（华南理工大学）
Grounded GUI Understanding for Vision-Based Spatial Intelligent Agent: Exemplified by Extended Reality Apps
The Chinese University of Hong Kong（香港中文大学）；Harbin Institute of Technology（哈尔滨工业大学）；Southern University of Science and Technology（南方科技大学）；The Hong Kong University of Science and Technology（香港科学大学）
MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representations
MedVisAI Lab（MedVisAI实验室）；ECE, Northwestern University（电子工程系，西北大学）；Institute for Infocomm Research (I 2 R), A*STAR, Singapore（信息与通信研究所（I2R），A*STAR，新加坡）；Lee Kong Chian School of Medicine, Nanyang Technological University（Lee Kong Chian医学院，南洋理工大学）
