超越叙事描述：通过多对抗训练从图像生成诗歌
Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training
摘要：
从图像中自动生成自然语言已经引起了广泛的关注。在本文中，我们采取一个步骤，进一步探讨自动诗歌创作的诗歌语言（多行）的图像生成。这项任务涉及多个挑战，包括从图像中发现诗意的线索（例如，绿色的希望），并生成诗歌，以满足两个相关性的图像和诗意的语言水平。为了解决上述问题，我们通过策略梯度，将诗歌生成任务分解为两个相互关联的子任务，通过多对抗训练，保证了跨模态关联性和诗歌语言风格。为了从图像中提取诗意的线索，我们建议学习一种深度耦合的视觉-诗意嵌入，在这种嵌入中，可以共同学习图像中的对象、情感和场景的诗意表示。进一步引入了两个判别网络来指导诗歌生成，包括多模态网络和诗歌风格网络。为了便于研究，我们发布了两个由人类注释者提供的诗歌数据集，它们具有两个不同的属性：1）第一个人类注释的图像到诗歌对数据集（总共有8，292对），以及2）迄今为止最大的公共英语诗歌语料库数据集（总共有92，265首不同的诗歌）。对8 K图像进行了广泛的实验，其中随机挑选了1.5K图像进行评估。客观和主观的评价都显示了优越的性能对国家的最先进的方法从图像生成诗歌。图灵测试进行了超过500人的主题，其中30名评估员是诗歌专家，证明了我们的方法的有效性。
英文摘要：
Automatic generation of natural language from images has attracted extensive attention. In this paper, we take one step further to investigate generation of poetic language (with multiple lines) to an image for automatic poetry creation. This task involves multiple challenges, including discovering poetic clues from the image (e.g., hope from green), and generating poems to satisfy both relevance to the image and poeticness in language level. To solve the above challenges, we formulate the task of poem generation into two correlated sub-tasks by multi-adversarial training via policy gradient, through which the cross-modal relevance and poetic language style can be ensured. To extract poetic clues from images, we propose to learn a deep coupled visual-poetic embedding, in which the poetic representation from objects, sentiments and scenes in an image can be jointly learned. Two discriminative networks are further introduced to guide the poem generation, including a multi-modal discriminator and a poem-style discriminator. To facilitate the research, we have released two poem datasets by human annotators with two distinct properties: 1) the first human annotated image-to-poem pair dataset (with 8,292 pairs in total), and 2) to-date the largest public English poem corpus dataset (with 92,265 different poems in total). Extensive experiments are conducted with 8K images, among which 1.5K image are randomly picked for evaluation. Both objective and subjective evaluations show the superior performances against the state-of-the-art methods for poem generation from images. Turing test carried out with over 500 human subjects, among which 30 evaluators are poetry experts, demonstrates the effectiveness of our approach.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
