arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1909.09070cs.AIcs.CLcs.CV
收藏

看,读,丰富。从科学人物及其标题中学习

Look, Read and Enrich. Learning from Scientific Figures and their Captions

Expert System Cogito Labs(Expert System Cogito实验室)

Jose Manuel Gomez-Perez, Raul Ortega

摘要:

与自然图像相比,理解科学数字对机器来说特别困难。然而,在科学文献中有一个有价值的信息来源,直到现在还没有被开发:一个数字和它的标题之间的对应关系。在本文中,我们研究了什么可以通过看大量的数字和阅读他们的标题,并介绍了一个数字标题对应学习任务,利用我们的观察。训练视觉和语言网络,而不是对不受约束的数字和标题的监督,成功地解决了这个任务。我们还表明,从知识图谱转移词汇和语义知识显着丰富的功能。最后,我们展示了这些功能在涉及科学文本和数字的其他任务中的积极影响,如多模态分类和机器理解问答,优于监督基线和ad-hoc方法。

英文摘要:

Compared to natural images, understanding scientific figures is particularly hard for machines. However, there is a valuable source of information in scientific literature that until now has remained untapped: the correspondence between a figure and its caption. In this paper we investigate what can be learnt by looking at a large number of figures and reading their captions, and introduce a figure-caption correspondence learning task that makes use of our observations. Training visual and language networks without supervision other than pairs of unconstrained figures and captions is shown to successfully solve this task. We also show that transferring lexical and semantic knowledge from a knowledge graph significantly enriches the resulting features. Finally, we demonstrate the positive impact of such features in other tasks involving scientific text and figures, like multi-modal classification and machine comprehension for question answering, outperforming supervised baselines and ad-hoc approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑