不要只是听，使用你的想象力：利用视觉常识进行非视觉任务
Don't Just Listen, Use Your Imagination: Leveraging Visual Common Sense for Non-Visual Tasks
摘要：
今天的人工智能可以回答事实问题。但他们在需要常识推理的问题上却做得不够。也许这是因为大多数现有的常识数据库依赖于文本来学习和表示知识。但很多常识知识都是不成文的--部分原因是这些知识往往不够有趣，不值得谈论，部分原因是有些常识不太自然，无法用文字表达出来。虽然没有写，但它不是看不见的。在本文中，我们利用从图像中学习到的语义常识知识-即视觉常识-在两个文本任务：填空和视觉释义。我们建议“想象”文本背后的场景，并利用“想象”的场景除了文本线索，同时回答这些问题的视觉线索。我们把这些场景想象成一种视觉抽象。我们的方法在这些任务上优于强大的纯文本基线。我们提出的任务可以作为基准，以定量评估解决任务的进展“面目全非”。我们的代码和数据集是公开的。
英文摘要：
Artificial agents today can answer factual questions. But they fall short on questions that require common sense reasoning. Perhaps this is because most existing common sense databases rely on text to learn and represent knowledge. But much of common sense knowledge is unwritten - partly because it tends not to be interesting enough to talk about, and partly because some common sense is unnatural to articulate in text. While unwritten, it is not unseen. In this paper we leverage semantic common sense knowledge learned from images - i.e. visual common sense - in two textual tasks: fill-in-the-blank and visual paraphrasing. We propose to "imagine" the scene behind the text, and leverage visual cues from the "imagined" scenes in addition to textual cues while answering these questions. We imagine the scenes as a visual abstraction. Our approach outperforms a strong text-only baseline on these tasks. Our proposed tasks can serve as benchmarks to quantitatively evaluate progress in solving tasks that go "beyond recognition". Our code and datasets are publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
