arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1709.01220cs.CV
收藏

用于大规模图像标注的多模态多尺度深度学习

Multi-Modal Multi-Scale Deep Learning for Large-Scale Image Annotation

Renmin University of China(中国人民大学);Queen Mary University of London(伦敦玛丽女王大学);Columbia University(哥伦比亚大学)

Yulei Niu, Zhiwu Lu, Ji-Rong Wen, Tao Xiang, Shih-Fu Chang

摘要:

图像注释旨在使用与不同视觉概念相对应的可变数量的类标签来注释给定图像。在本文中,我们解决了大规模图像标注中的两个主要问题:1)如何学习适合于预测从对象,场景到抽象概念的各种视觉概念的丰富特征表示; 2)如何用最佳数量的类标签来标注图像。为了解决第一个问题,我们提出了一种新的多尺度深度模型,用于提取能够代表广泛视觉概念的丰富和有区别的特征。具体地,提出了一种新的两分支深度神经网络架构,其包括非常深的主网络分支和被设计用于融合从主分支计算的多尺度特征的伴随特征融合网络分支。深度模型还通过将嘈杂的用户提供的标签作为模型输入以补充图像输入而成为多模态的。为了解决第二个问题,我们在主标签预测任务中引入了一个标签数量预测辅助任务,以明确估计给定图像的最佳标签数量。在两个大规模的图像标注基准数据集上进行了大量的实验,结果表明,我们的方法显着优于最先进的。

英文摘要:

Image annotation aims to annotate a given image with a variable number of class labels corresponding to diverse visual concepts. In this paper, we address two main issues in large-scale image annotation: 1) how to learn a rich feature representation suitable for predicting a diverse set of visual concepts ranging from object, scene to abstract concept; 2) how to annotate an image with the optimal number of class labels. To address the first issue, we propose a novel multi-scale deep model for extracting rich and discriminative features capable of representing a wide range of visual concepts. Specifically, a novel two-branch deep neural network architecture is proposed which comprises a very deep main network branch and a companion feature fusion network branch designed for fusing the multi-scale features computed from the main branch. The deep model is also made multi-modal by taking noisy user-provided tags as model input to complement the image input. For tackling the second issue, we introduce a label quantity prediction auxiliary task to the main label prediction task to explicitly estimate the optimal label number for a given image. Extensive experiments are carried out on two large-scale image annotation benchmark datasets and the results show that our method significantly outperforms the state-of-the-art.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑