arXiv 2504.041582025-04-05cs.CV

JarvisIR：以智能图像复原提升自动驾驶感知

JarvisIR: Elevating Autonomous Driving Perception with Intelligent Image Restoration

Xiamen University（厦门大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Bytedance’s Pico（字节跳动Pico）；Tencent（腾讯）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；The Chinese University of Hong Kong（香港中文大学）

Yunlong Lin, Zixu Lin, Haoyu Chen, Panwang Pan, Chenxin Li, Sixiang Chen, Yeying Jin, Wenbo Li, Xinghao Ding

备注：: 25 pages, 15 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

JarvisIR提出用视觉语言模型作为控制器调度多个专家复原模型，并通过监督微调与人类反馈对齐的两阶段框架提升真实恶劣天气下的感知鲁棒性，在CleanBench-Real上实现感知指标平均提升50%。

摘要：

以视觉为中心的感知系统在真实环境中难以应对不可预测且相互耦合的天气退化。当前解决方案往往存在局限，要么依赖特定的退化先验，要么存在显著的领域差距。为了在真实世界条件下实现稳健且自主的运行，我们提出JarvisIR，一个由视觉语言模型（VLM）驱动的智能体，它利用VLM作为控制器来管理多个专家复原模型。为了进一步增强系统稳健性、减少幻觉并提高在真实世界恶劣天气中的泛化能力，JarvisIR采用了一种新颖的两阶段框架，包括监督微调和人类反馈对齐。具体而言，为了解决真实场景中配对数据缺乏的问题，人类反馈对齐使VLM能够以无监督方式在大规模真实世界数据上得到有效微调。为了支持JarvisIR的训练和评估，我们引入了CleanBench，一个由高质量、大规模指令-响应对组成的综合数据集，包括150K条合成数据和80K条真实数据。大量实验表明，JarvisIR展现出优越的决策和复原能力。与现有方法相比，它在CleanBench-Real上的所有感知指标平均值上实现了50%的提升。项目页面：https://cvpr2025-jarvisir.github.io/。

英文摘要：

Vision-centric perception systems struggle with unpredictable and coupled weather degradations in the wild. Current solutions are often limited, as they either depend on specific degradation priors or suffer from significant domain gaps. To enable robust and autonomous operation in real-world conditions, we propose JarvisIR, a VLM-powered agent that leverages the VLM as a controller to manage multiple expert restoration models. To further enhance system robustness, reduce hallucinations, and improve generalizability in real-world adverse weather, JarvisIR employs a novel two-stage framework consisting of supervised fine-tuning and human feedback alignment. Specifically, to address the lack of paired data in real-world scenarios, the human feedback alignment enables the VLM to be fine-tuned effectively on large-scale real-world data in an unsupervised manner. To support the training and evaluation of JarvisIR, we introduce CleanBench, a comprehensive dataset consisting of high-quality and large-scale instruction-responses pairs, including 150K synthetic entries and 80K real entries. Extensive experiments demonstrate that JarvisIR exhibits superior decision-making and restoration capabilities. Compared with existing methods, it achieves a 50% improvement in the average of all perception metrics on CleanBench-Real. Project page: https://cvpr2025-jarvisir.github.io/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。