UniBoost：用于提升零样本视觉-语言任务的无监督单模态预训练
UniBoost: Unsupervised Unimodal Pre-training for Boosting Zero-shot Vision-Language Tasks
AI总结：
本文提出 UniBoost，利用大规模无监督单模态预训练增强图文模型表示能力，在零样本 PASCAL-5$^i$ 与 COCO-20$^i$ 分割上显著超越基于 CLIP 的方法。
摘要：
多模态模型（例如 CLIP）的大规模联合训练已在许多视觉-语言任务中展现出优异性能。然而，用于预训练的图文对被限制在图像与文本的交集内，限制了其覆盖真实世界数据广泛分布的能力；在预处理过程中，还可能因未对齐的样本对而引入噪声。相反，仅基于文本或图像数据、通过无监督技术训练的单模态模型，能够更广泛地覆盖多样化的真实世界数据，并且不受图像和文本必须同时存在这一要求的约束。本文证明，使用大规模无监督单模态模型作为预训练，可以增强图文对模型的零样本性能。我们的充分研究验证了，以此方式预训练的模型能够学习两种模态的丰富表示，提升其理解图像与文本之间关联的能力。实验表明，在零样本设置下，单模态预训练在 PASCAL-5$^i$ 语义分割上比基于 CLIP 的最先进模型高出 6.5%（52.3% $\ ightarrow$ 58.8%），在 COCO-20$^i$ 语义分割上高出 6.2%（27.2% $\ ightarrow$ 33.4%）。通过学习两种模态的表示，单模态预训练提供了更广的覆盖范围、更少的未对齐错误，并能够捕捉真实世界数据中更复杂的特征和模式，从而带来更好的性能，尤其适用于零样本视觉-语言任务。
英文摘要：
Large-scale joint training of multimodal models, e.g., CLIP, have demonstrated great performance in many vision-language tasks. However, image-text pairs for pre-training are restricted to the intersection of images and texts, limiting their ability to cover a large distribution of real-world data, where noise can also be introduced as misaligned pairs during pre-processing. Conversely, unimodal models trained on text or image data alone through unsupervised techniques can achieve broader coverage of diverse real-world data and are not constrained by the requirement of simultaneous presence of image and text. In this paper, we demonstrate that using large-scale unsupervised unimodal models as pre-training can enhance the zero-shot performance of image-text pair models. Our thorough studies validate that models pre-trained as such can learn rich representations of both modalities, improving their ability to understand how images and text relate to each other. Our experiments show that unimodal pre-training outperforms state-of-the-art CLIP-based models by 6.5% (52.3% $\rightarrow$ 58.8%) on PASCAL-5$^i$ and 6.2% (27.2% $\rightarrow$ 33.4%) on COCO-20$^i$ semantic segmentation under zero-shot setting respectively. By learning representations of both modalities, unimodal pre-training offers broader coverage, reduced misalignment errors, and the ability to capture more complex features and patterns in the real-world data resulting in better performance especially for zero-shot vision-language tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
