Instruction following is crucial in contemporary LLM. However, when extended to multimodal setting, it often suffers from misalignment between specific textual instruction and targeted local region of an image. To achieve more accurate and nuanced multimodal instruction following, we introduce Instruction-guided Visual Masking (IVM), a new versatile visual grounding model that is compatible with diverse multimodal models, such as LMM and robot model. By constructing visual masks for instruction-irrelevant regions, IVM-enhanced multimodal models can effectively focus on task-relevant image regions to better align with complex instructions. Specifically, we design a visual masking data generation pipeline and create an IVM-Mix-1M dataset with 1 million image-instruction pairs. We further introduce a new learning technique, Discriminator Weighted Supervised Learning (DWSL) for preferential IVM training that prioritizes high-quality data samples. Experimental results on generic multimodal tasks such as VQA and embodied robotic control demonstrate the versatility of IVM, which as a plug-and-play tool, significantly boosts the performance of diverse multimodal models, yielding new state-of-the-art results across challenging multimodal benchmarks. Code, model and data are available at https://github.com/2toinf/IVM.
iGVLM: Dynamic Instruction-Guided Vision Encoding for Question-Aware Multimodal Understanding
School of Computer Science（计算机科学学院）；Technology, Huazhong University of Science（科技，华中科技大学）；Li Auto Inc.（Li汽车公司）；Institute of AI for Industries, Chinese Academy of Sciences（产业人工智能研究所，中国科学院）
DiZiNER: Disagreement-guided Instruction Refinement via Pilot Annotation Simulation for Zero-shot Named Entity Recognition
Seltasquare Seoul, Korea（首尔韩国Seltasquare）；Seoul National University College of Medicine（首尔国立大学医学院）；Biomedical Research Institute, Seoul National University Hospital（首尔国立大学医院生物医学研究所）
Instruction-Guided Editing Controls for Images and Multimedia: A Survey in LLM era
Griffith University（格里菲斯大学）；University of Bremen（不来梅大学）；Ho Chi Minh City University of Technology（胡志明市技术大学）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；Hanoi University of Science and Technology（河内科技大学）；The University of Queensland（昆士兰大学）
Mitigating Cross-Modal Distraction and Ensuring Geometric Feasibility via Affordance-Guided and Self-Consistent MLLMs for Task Planning in Instruction-Following Manipulation
Department of Computer Science, National Yang Ming Chiao Tung University（计算机科学系，阳明交通大学）；Department of Computer Science and Information Engineering, National Taiwan University（计算机科学与信息工程系，台湾大学）
TableDreamer: Progressive and Weakness-guided Data Synthesis from Scratch for Table Instruction Tuning
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Baidu Inc（百度公司）
Beihang University（北京航空航天大学）；Xiaohongshu Inc（小红书公司）；Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Nanchang Institute of Technology（南昌工程学院）；Zhongguancun Laboratory（中关村实验室）
ProSwitch: Knowledge-Guided Instruction Tuning to Switch Between Professional and Non-Professional Responses
Zhejiang University of Science and Technology（浙江科技大学）；Fudan University（复旦大学）；Zhejiang University（浙江大学）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）
LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis
University of Chinese Academy of Sciences（中国科学院大学）；The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所认知与决策智能复杂系统重点实验室）
SleepWalk: A Three-Tier Benchmark for Stress-Testing Instruction-Guided Vision-Language Navigation
Indian AI Research Organization (IAIRO)（印度人工智能研究组织）；ĸragya Lab, BITS Pilani Goa（BITS Pilani Goa 的 ĸragya 实验室）；University of Dhaka（达卡大学）；Delhi Technological University（德里技术大学）；Apple（苹果公司）；Meta
Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；JD Explore Academy（京东探索研究院）；Renmin University of China（中国人民大学）
Recently, significant progress has been made in masked image modeling to catch up to masked language modeling. However, unlike words in NLP, the lack of semantic decomposition of images still makes masked autoencoding (MAE) different between vision and language. In this paper, we explore a potential visual analogue of words, i.e., semantic parts, and we integrate semantic information into the training process of MAE by proposing a Semantic-Guided Masking strategy. Compared to widely adopted random masking, our masking strategy can gradually guide the network to learn various information, i.e.,…
Beyond Image-Text Matching: Verb Understanding in Multimodal Transformers Using Guided Masking
Faculty of Information Technology, Brno University of Technology（布尔诺理工大学信息技术学院）；Kempelen Institute of Intelligent Technologies（肯佩伦智能技术研究所）；George Mason University（乔治梅森大学）
National Technical University of Athens（雅典国立技术大学）；Institute of Advanced Research in Artificial Intelligence (IARAI)（人工智能高级研究院（IARAI））；Athena RC（雅典娜研究中心）；University of Crete（克里特大学）；IACM-Forth（希腊计算机科学研究所-基础研究与技术基金会）
Transformers and masked language modeling are quickly being adopted and explored in computer vision as vision transformers and masked image modeling (MIM). In this work, we argue that image token masking differs from token masking in text, due to the amount and correlation of tokens in an image. In particular, to generate a challenging pretext task for MIM, we advocate a shift from random masking to informed masking. We develop and exhibit this idea in the context of distillation-based MIM, where a teacher transformer encoder generates an attention map, which we use to guide masking for the st…
Granular Ball Guided Masking: Structure-aware Data Augmentation
Chongqing Key Laboratory of Computational Intelligence（重庆计算智能重点实验室）；Key Laboratory of Cyberspace Big Data Intelligent Security（网络大数据智能安全重点实验室）；Sichuan-Chongqing Co-construction Key Laboratory of Digital Economy Intelligence（四川-重庆数字经济智能联合重点实验室）；Key Laboratory of Big Data Intelligent Computing（大数据智能计算重点实验室）；Chongqing University of Posts and Telecommunications（重庆邮电大学）
Physics-guided deep learning (PG-DL) via algorithm unrolling has received significant interest for improved image reconstruction, including MRI applications. These methods unroll an iterative optimization algorithm into a series of regularizer and data consistency units. The unrolled networks are typically trained end-to-end using a supervised approach. Current supervised PG-DL approaches use all of the available sub-sampled measurements in their data consistency units. Thus, the network learns to fit the rest of the measurements. In this study, we propose to improve the performance and robust…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；The University of Sydney（悉尼大学）；SenseTime（商汤科技）；Chinese Academy of Sciences - Artificial Intelligence Research（中国科学院人工智能创新研究院）
Semantic segmentation has achieved huge progress via adopting deep Fully Convolutional Networks (FCN). However, the performance of FCN based models severely rely on the amounts of pixel-level annotations which are expensive and time-consuming. To address this problem, it is a good choice to learn to segment with weak supervision from bounding boxes. How to make full use of the class-level and region-level supervisions from bounding boxes is the critical challenge for the weakly supervised learning task. In this paper, we first introduce a box-driven class-wise masking model (BCM) to remove irr…
East China Normal University（华东师范大学）；Xiaohongshu Inc.（小红书）；KLATASDS, MOE, China（中国教育部统计与数据科学重点实验室）；The Chinese University of Hong Kong（香港中文大学）；Zhejiang University（浙江大学）；Fudan University（复旦大学）；University of Oxford（牛津大学）；Shanghai Jiao Tong University（上海交通大学）；University of Science and Technology of China（中国科学技术大学）；Nanjing University（南京大学）
