arXiv 2502.017852025-02-03cs.CVcs.AI

AquaticCLIP：面向水下场景分析的视觉-语言基础模型

AquaticCLIP: A Vision-Language Foundation Model for Underwater Scene Analysis

Khalifa University of Science and Technology（哈利法科技大学）；Information Technology University（信息技术大学）

Basit Alawode, Iyyakutti Iyappan Ganapathi, Sajid Javed, Naoufel Werghi, Mohammed Bennamoun, Arif Mahmood

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出AquaticCLIP，一种基于200万对水下图像-文本数据训练的无监督对比学习视觉-语言基础模型，通过提示引导视觉编码器和视觉引导语言编码器实现多模态对齐，在零样本水下视觉任务中显著优于现有方法。

摘要：

保护水生生物多样性对于缓解气候变化的影响至关重要。水下场景理解在辅助海洋科学家进行决策过程中发挥着关键作用。本文提出了AquaticCLIP，这是一种专为水下场景理解设计的新型对比语言-图像预训练模型。AquaticCLIP提出了一种新的无监督学习框架，对水下环境中的图像和文本进行对齐，从而支持分割、分类、检测和物体计数等任务。通过利用我们大规模的水下图像-文本配对数据集，无需真实标注，我们的模型丰富了水生领域中现有的视觉-语言模型。为此，我们利用YouTube、Netflix、NatGeo等异构资源构建了一个包含200万对水下图像-文本配对的数据集。为了微调AquaticCLIP，我们提出了一种提示引导的视觉编码器，通过可学习提示逐步聚合图像块特征，同时一种视觉引导机制通过引入视觉上下文来增强语言编码器。该模型通过对比预训练损失进行优化，以对齐视觉和文本模态。AquaticCLIP在多个水下计算机视觉任务的零样本设置中取得了显著的性能提升，在鲁棒性和可解释性方面均优于现有方法。我们的模型为水下环境中的视觉-语言应用设立了新的基准。AquaticCLIP的代码和数据集已在GitHub上公开，地址为xxx。

英文摘要：

The preservation of aquatic biodiversity is critical in mitigating the effects of climate change. Aquatic scene understanding plays a pivotal role in aiding marine scientists in their decision-making processes. In this paper, we introduce AquaticCLIP, a novel contrastive language-image pre-training model tailored for aquatic scene understanding. AquaticCLIP presents a new unsupervised learning framework that aligns images and texts in aquatic environments, enabling tasks such as segmentation, classification, detection, and object counting. By leveraging our large-scale underwater image-text paired dataset without the need for ground-truth annotations, our model enriches existing vision-language models in the aquatic domain. For this purpose, we construct a 2 million underwater image-text paired dataset using heterogeneous resources, including YouTube, Netflix, NatGeo, etc. To fine-tune AquaticCLIP, we propose a prompt-guided vision encoder that progressively aggregates patch features via learnable prompts, while a vision-guided mechanism enhances the language encoder by incorporating visual context. The model is optimized through a contrastive pretraining loss to align visual and textual modalities. AquaticCLIP achieves notable performance improvements in zero-shot settings across multiple underwater computer vision tasks, outperforming existing methods in both robustness and interpretability. Our model sets a new benchmark for vision-language applications in underwater environments. The code and dataset for AquaticCLIP are publicly available on GitHub at xxx.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。