利用句子嵌入扩展多领域语义分割
Scaling up Multi-domain Semantic Segmentation with Sentence Embeddings
AI总结：
本文提出用类别描述句子的向量嵌入替代类别标签，实现零样本语义分割，并通过合并超200万张图像的多领域数据集训练模型，在7个基准上达到监督方法水平，微调后超越NYUD-V2和PASCAL-context最先进结果。
摘要：
我们提出了一种语义分割方法，在零样本设置下应用时可实现最先进的监督性能。因此，它在每个主要语义分割数据集上都能取得与监督方法相当的结果，而无需在这些数据集上进行训练。这是通过将每个类别标签替换为描述该类别的短段落的向量值嵌入来实现的。该方法的通用性和简洁性使得能够合并来自不同领域的多个数据集，每个数据集具有不同的类别标签和语义。由此产生的超过200万张图像的合并语义分割数据集，使得训练出的模型在7个基准数据集上取得了与最先进监督方法相当的性能，尽管没有使用其中的任何图像。通过在标准语义分割数据集上微调该模型，我们还在NYUD-V2和PASCAL-context上分别以60%和65%的mIoU显著超越了最先进的监督分割方法。基于语言嵌入的接近程度，我们的方法甚至可以分割未见过的标签。大量实验证明了该方法对未见图像领域和未见标签的强大泛化能力，并且该方法在下游应用中实现了令人印象深刻的性能提升，包括深度估计和实例分割。
英文摘要：
We propose an approach to semantic segmentation that achieves state-of-the-art supervised performance when applied in a zero-shot setting. It thus achieves results equivalent to those of the supervised methods, on each of the major semantic segmentation datasets, without training on those datasets. This is achieved by replacing each class label with a vector-valued embedding of a short paragraph that describes the class. The generality and simplicity of this approach enables merging multiple datasets from different domains, each with varying class labels and semantics. The resulting merged semantic segmentation dataset of over 2 Million images enables training a model that achieves performance equal to that of state-of-the-art supervised methods on 7 benchmark datasets, despite not using any images therefrom. By fine-tuning the model on standard semantic segmentation datasets, we also achieve a significant improvement over the state-of-the-art supervised segmentation on NYUD-V2 and PASCAL-context at 60% and 65% mIoU, respectively. Based on the closeness of language embeddings, our method can even segment unseen labels. Extensive experiments demonstrate strong generalization to unseen image domains and unseen labels, and that the method enables impressive performance improvements in downstream applications, including depth estimation and instance segmentation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
