In this paper, we present the Draw-and-Understand framework, exploring how to integrate visual prompting understanding capabilities into Multimodal Large Language Models (MLLMs). Visual prompts allow users to interact through multi-modal instructions, enhancing the models' interactivity and fine-grained image comprehension. In this framework, we propose a general architecture adaptable to different pre-trained MLLMs, enabling it to recognize various types of visual prompts (such as points, bounding boxes, and free-form shapes) alongside language understanding. Additionally, we introduce MDVP-Instruct-Data, a multi-domain dataset featuring 1.2 million image-visual prompt-text triplets, including natural images, document images, scene text images, mobile/web screenshots, and remote sensing images. Building on this dataset, we introduce MDVP-Bench, a challenging benchmark designed to evaluate a model's ability to understand visual prompting instructions. The experimental results demonstrate that our framework can be easily and effectively applied to various MLLMs, such as SPHINX-X and LLaVA. After training with MDVP-Instruct-Data and image-level instruction datasets, our models exhibit impressive multimodal interaction capabilities and pixel-level understanding, while maintaining their image-level visual perception performance.
Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；VCIP & TMCC & DISSec, College of Computer Science, Nankai University（南开大学计算机学院）；Department of Psychological and Cognitive Sciences, Tsinghua University（清华大学心理学与认知科学系）；University of Chinese Academy of Sciences（中国科学院大学）
What Large Language Models Bring to Text-rich VQA?
SenseTime Research（商汤科技研究院）；Nanjing University of Science and Technology（南京理工大学）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；ByteDance Inc.（字节跳动有限公司）
PunchBench: Benchmarking MLLMs in Multimodal Punchline Comprehension
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机学院，北京大学）；WeChat AI, Tencent Inc., China（微信AI，腾讯公司，中国）
MathScape: Benchmarking Multimodal Large Language Models in Real-World Mathematical Contexts
Peking University（北京大学）；University of Chinese Academy of Sciences（中国科学院大学）；Nankai University（南开大学）；Beijing Institute of Technology（北京理工大学）；Baichuan Inc.（百度文心）
Fudan University（复旦大学）；OpenGVLab（开放视觉实验室）；Shanghai AI Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）；The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；Nanjing University（南京大学）；SenseTime Research（商汤科技研究院）
CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models
Tsinghua University（清华大学）；Jilin University（吉林大学）；Northwest Minzu University（西北民族大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
ComprehendEdit: A Comprehensive Dataset and Evaluation Framework for Multimodal Knowledge Editing
Harbin Institute of Technology（哈尔滨工业大学）；Pengcheng Laboratory（鹏城实验室）；Shenzhen University of Advanced Technology（深圳先进技术大学）；Northwestern Polytechnical University（西北工业大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model
University of Washington（华盛顿大学）；Tsinghua University（清华大学）；Tencent（腾讯）；Google Deepmind（谷歌DeepMind）；Allen Institute for AI（艾伦人工智能研究所）；Cornell University（康奈尔大学）
TAMMs: Change Understanding and Forecasting in Satellite Image Time Series with Temporal-Aware Multimodal Models
School of Computer Science & Technology, Beijing Institute of Technology（计算机科学与技术学院，北京理工大学）；School of Computing, National University of Singapore（计算学院，新加坡国立大学）
DIP-R1: Deep Inspection and Perception with RL Looking Through and Understanding Complex Scenes
Integrated Vision and Language Lab., School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST)（整合视觉与语言实验室，电气工程学院，韩国科学技术院（KAIST））
KptLLM: Unveiling the Power of Large Language Model for Keypoint Comprehension
Sun Yat-sen University（中山大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）
From Image to Video, what do we need in multimodal LLMs?
Intelligent Computing and Machine Learning Lab, School of ASEE, Beihang University（北京航空航天大学自动化学院智能计算与机器学习实验室）；Xiaohongshu（小红书）；School of Sino-French Engineer, Beihang University（北京航空航天大学中法工程师学院）；College of Engineering and Computer Science, VinUniversity（Vin大学工程与计算机科学学院）
3DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding
Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing Digital Native Digital City Research Center（北京数字原生数字城市研究中心）；Peking University（北京大学）；Beihang University（北京航空航天大学）；Beijing University of Science and Technology（北京科技大学）
v-HUB: A Benchmark for Video Humor Understanding from Vision and Sound
Shanghai Jiao Tong University（上海交通大学）；Wuhan University（武汉大学）；Beijing Institute for General Artificial Intelligence（北京一般人工智能研究院）；Independent Researcher（独立研究者）
MUVR: A Multi-Modal Untrimmed Video Retrieval Benchmark with Multi-Level Visual Correspondence
MoE Key Laboratory of Brain-Machine Intelligence Technology, College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics（脑机智能技术MoE实验室，人工智能学院，南京航空航天大学）；Nanjing University（南京大学）；The Hong Kong Polytechnic University（香港理工大学）
Alpha-CLIP: A CLIP Model Focusing on Wherever You Want
Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；University of Macau（澳门大学）；MThreads, Inc.（摩尔线程公司）；Shanghai Jiao Tong University（上海交通大学）；Fudan University（复旦大学）
