通知：联邦节假日调休：哥伦布日 / 原住民日（Columbus Day / Indigenous Peoples' Day），今日（13日）无更新。通知：联邦节假日调休：哥伦布日 / 原住民日（Columbus Day / Indigenous Peoples' Day），今日（13日）无更新。
arXiv 2411.04923cs.CV
VideoGLaMM：一种用于视频像素级视觉定位的大型多模态模型
VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Tianjin University（天津大学）；Carnegie Mellon University（卡内基梅隆大学）；Linköping University（林雪平大学）；Australian National University（澳大利亚国立大学）
Shehan Munasinghe, Hanan Gani, Wenqi Zhu, Jiale Cao, Eric Xing, Fahad Shahbaz Khan, Salman Khan
Fine-grained alignment between videos and text is challenging due to complex spatial and temporal dynamics in videos. Existing video-based Large Multimodal Models (LMMs) handle basic conversations but struggle with precise pixel-level grounding in videos. To address this, we introduce VideoGLaMM, a LMM designed for fine-grained pixel-level grounding in videos based on user-provided textual inputs. Our design seamlessly connects three key components: a Large Language Model, a dual vision encoder that emphasizes both spatial and temporal details, and a spatio-temporal decoder for accurate mask generation. This connection is facilitated via tunable V-L and L-V adapters that enable close Vision-Language (VL) alignment. The architecture is trained to synchronize both spatial and temporal elements of video content with textual instructions. To enable fine-grained grounding, we curate a multimodal dataset featuring detailed visually-grounded conversations using a semiautomatic annotation pipeline, resulting in a diverse set of 38k video-QA triplets along with 83k objects and 671k masks. We evaluate VideoGLaMM on three challenging tasks: Grounded Conversation Generation, Visual Grounding, and Referring Video Segmentation. Experimental results show that our model consistently outperforms existing approaches across all three tasks.
MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output
School of Software & Microelectronics, Peking University（软件与微电子学院，北京大学）；School of Computer Science, Peking University（计算机学院，北京大学）；National Engineering Research Center for Software Engineering, Peking University（软件工程国家工程研究中心，北京大学）；Peking University Sixth Hospital（北京大学第六医院）；Augusta University（奥古斯塔大学）；Peking University First Hospital（北京大学第一医院）
EARL: Towards a Unified Analysis-Guided Reinforcement Learning Framework for Egocentric Interaction Reasoning and Pixel Grounding
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong SAR（香港理工大学电子与电气工程系）；Division of Emerging Interdisciplinary Areas (EMIA), The Hong Kong University of Science and Technology, Hong Kong SAR（香港理工大学新兴跨学科领域研究中心）
Institute of High Performance Computing, A*STAR（A*STAR高性能计算研究所）；Nanyang Technological University（南洋理工大学）
Current one-stage methods for visual grounding encode the language query as one holistic sentence embedding before fusion with visual feature. Such a formulation does not treat each word of a query sentence on par when modeling language to visual attention, therefore prone to neglect words which are less important for sentence embedding but critical for visual grounding. In this paper we propose Word2Pix: a one-stage visual grounding network based on encoder-decoder transformer architecture that enables learning for textual to visual feature correspondence via word to pixel attention. The embe…
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Australian National University（澳大利亚国立大学）；Aalto University（阿尔托大学）；Carnegie Mellon University（卡内基梅隆大学）；University of California - Merced（加州大学默塞德分校）；Linköping University（林雪平大学）；Google Research（谷歌研究院）
Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level
University of Central Florida（中佛罗里达大学）；University of Western Australia（西澳大学）；Amazon Web Services（亚马逊网络服务）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）
On the Design Fundamentals of Pixel Text Representation Learning
The Chinese University of Hong Kong（香港中文大学）；DAMO Academy, Alibaba Group（阿里巴巴达摩院）；Fudan University（复旦大学）；Aberystwyth University（阿伯里斯特威斯大学）；University of Macau（澳门大学）；Shanghai University of Finance and Economics（上海财经大学）
Arizona State University（亚利桑那州立大学）；University of California, Irvine（加州大学欧文分校）
Computer Vision applications often require a textual grounding module with precision, interpretability, and resilience to counterfactual inputs/queries. To achieve high grounding precision, current textual grounding methods heavily rely on large-scale training data with manual annotations at the pixel level. Such annotations are expensive to obtain and thus severely narrow the model's scope of real-world applications. Moreover, most of these methods sacrifice interpretability, generalizability, and they neglect the importance of being resilient to counterfactual inputs. To address these issues…
Improving GUI Grounding with Explicit Position-to-Coordinate Mapping
ServiceNow；Mila - Quebec AI Institute（魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；York University（约克大学）；Polytechnique Montréal（蒙特利尔理工学院）；McGill University（麦吉尔大学）
PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection
Inria, École normale supérieure, CNRS, PSL Research University（法国国家信息与自动化研究所，巴黎高等师范学院，法国国家科学研究中心，巴黎文理研究大学）；Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague（捷克信息学、机器人与控制论研究所，布拉格捷克理工大学）
DDVT: Dynamic Dual-level Vision Transformer Fusion Network for Answer Grounding in Visual Question Answering
National and Local Joint Engineering Laboratory of Computer Aided Design, School of Software Engineering, Dalian University（大连大学软件工程学院计算机辅助设计国家地方联合工程实验室）；School of Computer Science and Technology, Dalian University of Technology（大连理工大学计算机科学与技术学院）
Pengcheng Laboratory（鹏城实验室）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
The University of Texas MD Anderson Cancer Center（德克萨斯大学MD安德森癌症中心）；Eisai Inc.（卫材株式会社）；IISc, Bangalore（印度科学研究所班加罗尔分校）；Cohere Labs Community（Cohere实验室社区）
GeoGround: A Unified Large Vision-Language Model for Remote Sensing Visual Grounding
Nanyang Technological University（南洋理工大学）；University of Reading（阅读大学）；Shanghai Jiao Tong University（上海交通大学）；Harbin Institute of Technology（哈尔滨工业大学）；SenseTime Research（商汤科技研究院）
Towards Unified Surgical Scene Understanding:Bridging Reasoning and Grounding via MLLMs
Southern University of Science and Technology（南方科技大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；Northwestern University（西北大学）；University of Alberta（阿尔伯塔大学）；Yale University（耶鲁大学）；Nanfang Hospital（南华医院）；Shenzhen University of Advanced Technology（深圳大学先进技术研究院）
