arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.01736cs.CVcs.AIcs.LG
收藏

DaTaSeg:驯服通用多数据集多任务分割模型

DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Model

Google Research(谷歌研究院);ByteDance(字节跳动)

Xiuye Gu, Yin Cui, Jonathan Huang, Abdullah Rashwan, Xuan Yang, Xingyi Zhou, Golnaz Ghiasi, Weicheng Kuo, Huizhong Chen, Liang-Chieh Chen, David A Ross

AI总结:

本文提出 DaTaSeg,通过共享掩码提议表示、语义文本嵌入和全参数共享,训练跨数据集、多任务的通用分割模型,并验证其性能提升、弱监督迁移、开放词汇与规模化能力。

摘要:

观察到全景分割、语义分割和实例分割任务之间的密切关系,我们提出训练一个通用的多数据集多任务分割模型:DaTaSeg。我们对所有任务使用共享表示(带有类别预测的掩码提议)。为解决任务差异,我们针对不同任务采用不同的合并操作和后处理。我们还利用弱监督,使分割模型能够从成本更低的边界框标注中获益。为跨数据集共享知识,我们使用来自同一语义嵌入空间的文本嵌入作为分类器,并在数据集之间共享所有网络参数。我们在 ADE semantic、COCO panoptic 和 Objects365 detection 数据集上训练 DaTaSeg。DaTaSeg 提升了所有数据集上的性能,尤其是小规模数据集,在 ADE semantic 上达到 54.0 mIoU,在 COCO panoptic 上达到 53.5 PQ。DaTaSeg 还支持在 ADE panoptic 和 Objects365 instance segmentation 上进行弱监督知识迁移。实验表明,DaTaSeg 随训练数据集数量增加而扩展,并可通过直接迁移实现开放词汇分割。此外,我们标注了一个包含 1,000 张图像的 Objects365 实例分割集,并将将其作为公开基准发布。

英文摘要:

Observing the close relationship among panoptic, semantic and instance segmentation tasks, we propose to train a universal multi-dataset multi-task segmentation model: DaTaSeg.We use a shared representation (mask proposals with class predictions) for all tasks. To tackle task discrepancy, we adopt different merge operations and post-processing for different tasks. We also leverage weak-supervision, allowing our segmentation model to benefit from cheaper bounding box annotations. To share knowledge across datasets, we use text embeddings from the same semantic embedding space as classifiers and share all network parameters among datasets. We train DaTaSeg on ADE semantic, COCO panoptic, and Objects365 detection datasets. DaTaSeg improves performance on all datasets, especially small-scale datasets, achieving 54.0 mIoU on ADE semantic and 53.5 PQ on COCO panoptic. DaTaSeg also enables weakly-supervised knowledge transfer on ADE panoptic and Objects365 instance segmentation. Experiments show DaTaSeg scales with the number of training datasets and enables open-vocabulary segmentation through direct transfer. In addition, we annotate an Objects365 instance segmentation set of 1,000 images and will release it as a public benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑