Understanding Degradation with Vision Language Model
School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University（人工智能学院、光学与电子学（iOPEN）、西北工业大学）；Honors College, Northwestern Polytechnical University（荣誉学院、西北工业大学）；Shanghai AI Laboratory（上海人工智能实验室）；School of Artificial Intelligence, Shanghai Jiaotong University（人工智能学院、上海交通大学）
Guanzhou Lan, Chenyi Liao, Yuqi Yang, Qianli Ma, Zhigang Wang, Dong Wang, Bin Zhao, Xuelong Li
Understanding visual degradations is a critical yet challenging problem in computer vision. While recent Vision-Language Models (VLMs) excel at qualitative description, they often fall short in understanding the parametric physics underlying image degradations. In this work, we redefine degradation understanding as a hierarchical structured prediction task, necessitating the concurrent estimation of degradation types, parameter keys, and their continuous physical values. Although these sub-tasks operate in disparate spaces, we prove that they can be unified under one autoregressive next-token prediction paradigm, whose error is bounded by the value-space quantization grid. Building on this insight, we introduce DU-VLM, a multimodal chain-of-thought model trained with supervised fine-tuning and reinforcement learning using structured rewards. Furthermore, we show that DU-VLM can serve as a zero-shot controller for pre-trained diffusion models, enabling high-fidelity image restoration without fine-tuning the generative backbone. We also introduce \textbf{DU-110k}, a large-scale dataset comprising 110,000 clean-degraded pairs with grounded physical annotations. Extensive experiments demonstrate that our approach significantly outperforms generalist baselines in both accuracy and robustness, exhibiting generalization to unseen distributions.
UniProcessor: A Text-induced Unified Low-level Image Processor
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University（上海交通大学图像通信与网络工程研究所）；MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院、教育部人工智能重点实验室）
Degradation-Aware Prompt Learning with Cross-Modal Compensation for Adverse Weather Removal
School of Software Engineering, Dalian University（大连大学软件工程学院）；School of Mathematical Sciences, Dalian University of Technology（大连理工大学数学科学学院）；The Hong Kong Polytechnic University（香港理工大学）；University of California, San Francisco（加利福尼亚大学旧金山分校）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）
Mitigating Visual Degradation in MLLMs via Spatial-Spectral Visual Anchor Learning
China University of Petroleum (East China)（中国石油大学（华东））；Shanghai Jiao Tong University（上海交通大学）；Wuhan University（武汉大学）；Great Wall Motor（长城汽车）；Nanyang Technological University（南洋理工大学）；The University of Hong Kong（香港大学）
VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework
Stanford University（斯坦福大学）；Peking University（北京大学）；Harvard University（哈佛大学）；University of Washington, Seattle（华盛顿大学西雅图分校）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the…
SpatialBot: Precise Spatial Understanding with Vision Language Models
School of Artificial Intelligence, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Stanford University（斯坦福大学）；BAAI（北京智源人工智能研究院）；Peking University（北京大学）；University of Oxford（牛津大学）；Southeast University（东南大学）
Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Research Center of Artificial Intelligence, Peng Cheng Laboratory（鹏城实验室人工智能研究中心）
Vision-language models can encode societal biases and stereotypes, but there are challenges to measuring and mitigating these multimodal harms due to lacking measurement robustness and feature degradation. To address these challenges, we investigate bias measures and apply ranking metrics for image-text representations. We then investigate debiasing methods and show that prepending learned embeddings to text queries that are jointly trained with adversarial debiasing and a contrastive loss reduces various bias measures with minimal degradation to the image-text representation.
Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation
Clear Roads, Clear Vision: Advancements in Multi-Weather Restoration for Smart Transportation
Finolex Academy of Management and Technology（Finolex管理与技术学院）；Drone Lab, Centre for Artificial Intelligence and Robotics, IIT Mandi（印度理工学院曼迪分校人工智能与机器人中心无人机实验室）；Mehta Family School of Data Science and Artificial Intelligence, IIT Guwahati（印度理工学院古瓦哈提分校梅塔家族数据科学与人工智能学院）；SPACE42；Centre of Excellence: Artificial Intelligence, School of Computer Science, UPES Dehradun（UPES德拉敦分校计算机科学学院人工智能卓越中心）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
America has a massive railway system. As of 2006, U.S. freight railroads have 140,490 route- miles of standard gauge, but maintaining such a huge system and eliminating any dangers, like reduced track stability and poor drainage, caused by railway ballast degradation require huge amount of labor. The traditional way to quantify the degradation of ballast is to use an index called Fouling Index (FI) through ballast sampling and sieve analysis. However, determining the FI values in lab is very time-consuming and laborious, but with the help of recent development in the field of computer vision, …
ControlFusion: A Controllable Image Fusion Framework with Language-Vision Degradation Prompts
Electronic Information School, Wuhan University（武汉大学电子信息学院）；School of Computer Science and Engineering, Macau University of Science and Technology（澳门科学技术大学计算机科学与工程学院）；Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）
University of California, Santa Barbara（加州大学圣塔芭芭拉分校）
Most existing vision-language pre-training methods focus on understanding tasks and use BERT-like objectives (masked language modeling and image-text matching) during pretraining. Although they perform well in many understanding downstream tasks, e.g., visual question answering, image-text retrieval and visual entailment, they do not possess the ability to generate. To tackle this problem, we propose Unified multimodal pre-training for both Vision-Language understanding and generation (UniVL). The proposed UniVL is capable of handling both understanding tasks and generative tasks. We augment e…
Harbin Institute of Technology（哈尔滨工业大学）；Microsoft Research（微软研究院）；Peng Cheng Laboratory（鹏城实验室）
We propose a cross-modal attention distillation framework to train a dual-encoder model for vision-language understanding tasks, such as visual reasoning and visual question answering. Dual-encoder models have a faster inference speed than fusion-encoder models and enable the pre-computation of images and text during inference. However, the shallow interaction module used in dual-encoder models is insufficient to handle complex vision-language understanding tasks. In order to learn deep interactions of images and text, we introduce cross-modal attention distillation, which uses the image-to-te…
