arXiv 2606.267942026-06-26cs.CVcs.AI

ReasonCLIP-58M: CLIP的视觉基础常识推理监督

ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP

Khalifa University（卡利法大学）；University of Western Australia（西澳大学）；The Chinese University of Hong Kong（香港中文大学）；University of Melbourne（墨尔本大学）；University of Central Florida（佛罗里达中央大学）

Sicheng Zhang, Muzammal Naseer, Binzhu Xie, Naufal Suryanto, Shi Qiu, Jamal Bentahar, Naveed Akhtar, Mubarak Shah

备注：: Accepted to ECCV2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出ReasonCLIP-58M框架，通过两阶段策略将大规模推理监督集成到CLIP模型中，构建ReasonLite-42M和ReasonPro-16M数据集及RCLIP-Bench基准，提升视觉基础常识推理和零样本检索性能。

摘要：

CLIP及其变体被广泛用作多模态系统中的视觉骨干网络，但其预训练仍以描述性图像-文本对齐为主。随着下游应用日益需要视觉基础的常识推理和组合推理，目前尚不清楚CLIP风格的编码器是否能在不改变架构的情况下支持此类推理。为解决这一问题，我们提出ReasonCLIP-58M，一种持续预训练框架，通过我们的两阶段策略将大规模推理监督集成到CLIP风格模型中，该策略逐步整合推理信号同时保留描述性对齐，随后进行类别结构化的推理监督。为支持该框架，我们构建了两个互补数据集和一个基准：ReasonLite-42M，包含开放形式、视觉可验证的推理描述；ReasonPro-16M，包含类别特定的推理监督；以及RCLIP-Bench，用于视觉基础推理的诊断评估。我们训练了一系列ReasonCLIP模型，这些模型在提升视觉基础常识和组合推理的同时，也增强了零样本检索性能。作为多模态大语言模型（如LLaVA-NeXT）的即插即用视觉编码器，ReasonCLIP在不增加推理成本的情况下带来持续改进，表明结构化推理监督增强了CLIP风格视觉表示的表达能力。所有数据集、模型和训练代码均可在该网址获取。

英文摘要：

CLIP and its variants are widely adopted visual backbones in multimodal systems, but their pretraining remains dominated by descriptive image-text alignment. As downstream applications increasingly demand visually grounded commonsense inference and compositional reasoning, it remains unclear whether CLIP-style encoders can support such reasoning without architectural changes. To address this, we present ReasonCLIP-58M, a continual pretraining framework that integrates large-scale reasoning supervision into CLIP-style models through our two-stage strategy, which progressively integrates reasoning signals while preserving descriptive alignment, followed by category-structured reasoning supervision. To support this framework, we construct two complementary datasets and a benchmark: ReasonLite-42M, with open-form, visually verifiable reasoning captions; ReasonPro-16M, with category-specific reasoning supervision; and RCLIP-Bench for diagnostic evaluation of visually grounded reasoning. We train a family of ReasonCLIP that improves visually grounded commonsense and compositional reasoning while also enhancing zero-shot retrieval performance. As a drop-in visual encoder for multimodal large language models such as LLaVA-NeXT, ReasonCLIP delivers consistent gains without additional inference cost, demonstrating that structured reasoning supervision enhances the expressive capacity of CLIP-style visual representations. All datasets, models, and training code are available at https://github.com/RISys-Lab/ReasonCLIP.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。