PowerCLIP: 基于幂集对齐的对比预训练
PowerCLIP: Powerset Alignment for Contrastive Pre-Training
AI总结：
PowerCLIP通过引入幂集对齐方法，提升多区域图像语义理解，实验表明其在零样本分类和检索任务中表现更优。
摘要：
对比视觉-语言预训练框架如CLIP在多种视觉-语言任务中展示了出色的零样本性能。近期研究表明，将单个文本标记与特定图像块或区域对齐可以增强细粒度组成理解。然而，捕捉跨越多个图像区域的组成语义仍具挑战性。为解决此限制，我们提出PowerCLIP，一种通过幂集对齐增强的新型对比预训练框架，通过最小化图像区域幂集与文本解析树之间的损失，全面优化区域到短语的对齐。由于朴素的幂集构造因区域子集数量的指数增长导致计算成本高，我们引入高效的非线性聚合器（NLAs），将复杂度从O(2^M)降低到O(M)，同时以任意精度近似精确损失值。我们的广泛实验表明，PowerCLIP在零样本分类和检索任务中优于现有最先进方法，凸显了我们方法的组成性和鲁棒性。代码可在https://github.com/Masakichi210/PowerCLIP获取。
英文摘要：
Contrastive vision-language pre-training frameworks such as CLIP have demonstrated impressive zero-shot performance across a range of vision-language tasks. Recent studies have shown that aligning individual text tokens with specific image patches or regions enhances fine-grained compositional understanding. However, it remains challenging to capture compositional semantics that span multiple image regions. To address this limitation, we propose PowerCLIP, a novel contrastive pre-training framework enhanced by powerset alignment, which exhaustively optimizes region-to-phrase alignments by minimizing the loss defined between powersets of image regions and textual parse trees. Since the naive powerset construction incurs exponential computational cost due to the combinatorial explosion in the number of region subsets, we introduce efficient non-linear aggregators (NLAs) that reduce complexity from O(2^M) to O(M) with respect to the number of regions M, while approximating the exact loss value with arbitrary precision. Our extensive experiments demonstrate that PowerCLIP outperforms state-of-the-art methods in zero-shot classification and retrieval tasks, underscoring the compositionality and robustness of our approach. Code is available at https://github.com/Masakichi210/PowerCLIP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
