arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1902.09969cs.CVcs.CLcs.LG
收藏

使用深层对象特征进行图像描述

Using Deep Object Features for Image Descriptions

Ashutosh Mishra, Marcus Liwicki

摘要:

受机器翻译中利用多种模态的最新进展的启发,我们引入了一个编码器-解码器流水线,该流水线使用(1)图像中的特定对象及其对象标签,(2)用于解码对象特征和对象标签的联合嵌入的语言模型。我们的流水线将先前从图像中检测到的对象及其对象标签合并,然后学习描述特定图像的标题序列。解码器模型通过解码由编码器组件调节的对象视觉特征及其对象类的联合表示来学习从零开始提取图像的描述。该模型的思想是只关注图像的特定对象及其标签,以生成图像的描述,而不是整个图像的视觉特征。模型需要通过调整参数和设置进行更多的校准,以获得更好的准确性和性能。

英文摘要:

Inspired by recent advances in leveraging multiple modalities in machine translation, we introduce an encoder-decoder pipeline that uses (1) specific objects within an image and their object labels, (2) a language model for decoding joint embedding of object features and the object labels. Our pipeline merges prior detected objects from the image and their object labels and then learns the sequences of captions describing the particular image. The decoder model learns to extract descriptions for the image from scratch by decoding the joint representation of the object visual features and their object classes conditioned by the encoder component. The idea of the model is to concentrate only on the specific objects of the image and their labels for generating descriptions of the image rather than visual feature of the entire image. The model needs to be calibrated more by adjusting the parameters and settings to result in better accuracy and performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑