arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.03432cs.CV
收藏

文本监督语义分割中多视图损失的SimCon

SimCon Loss with Multiple Views for Text Supervised Semantic Segmentation

Czech Technical University in Prague(布拉格捷克理工大学);AWS AI Labs(AWS人工智能实验室)

Yash Patel, Yusheng Xie, Yi Zhu, Srikar Appalaraju, R. Manmatha

摘要:

纯粹依靠来自web数据的图文对齐来学习分割图像会由于数据中的噪声而导致次优性能。噪声来自相关文本与图像的视觉内容不相关的样本。本文提出了一种新的损失函数SimCon,而不是纯粹依赖于噪声数据的对齐,该函数考虑了模态内的相似性,以确定要对齐的适当的正样本集。此外,使用图像的多个视图(合成创建)进行训练并将SimCon损失与之结合,使训练更加稳健。这个版本的损失被称为MV-SimCon。实验结果表明,使用所提出的损失函数导致一致的改进zero-shot,文本监督语义分割和性能优于最先进的$+3.0\%$,$+3.3\%$和$+6.9\%$ PASCAL VOC,PASCAL上下文和MSCOCO,分别。随着测试时间的增加,我们通过将这些结果分别改进为PASCAL VOC、PASCAL Context和MSCOCO上的$58.7\%$、$26.6\%$和$33.3\%$,从而创造了新的记录。此外,使用所提出的损失函数导致鲁棒的训练和更快的收敛。

英文摘要:

Learning to segment images purely by relying on the image-text alignment from web data can lead to sub-optimal performance due to noise in the data. The noise comes from the samples where the associated text does not correlate with the image's visual content. Instead of purely relying on the alignment from the noisy data, this paper proposes a novel loss function termed SimCon, which accounts for intra-modal similarities to determine the appropriate set of positive samples to align. Further, using multiple views of the image (created synthetically) for training and combining the SimCon loss with it makes the training more robust. This version of the loss is termed MV-SimCon. The empirical results demonstrate that using the proposed loss function leads to consistent improvements on zero-shot, text supervised semantic segmentation and outperforms state-of-the-art by $+3.0\%$, $+3.3\%$ and $+6.9\%$ on PASCAL VOC, PASCAL Context and MSCOCO, respectively. With test time augmentations, we set a new record by improving these results further to $58.7\%$, $26.6\%$, and $33.3\%$ on PASCAL VOC, PASCAL Context, and MSCOCO, respectively. In addition, using the proposed loss function leads to robust training and faster convergence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑