arXiv 2209.101342024-02-18cs.MMcs.CLcs.CV

从未分割烹饪视频中生成食谱

Recipe Generation from Unsegmented Cooking Videos

Graduate School of Informatics, Kyoto University（京都大学情报学研究科）；OMRON SINIC X Corporation（欧姆龙 SINIC X 公司）；Academic Center for Computing and Media Studies, Kyoto University（京都大学信息与媒体学术中心）

Taichi Nishimura, Atsushi Hashimoto, Yoshitaka Ushiku, Hirotaka Kameko, Shinsuke Mori

备注：: Accepted at ACM TOMM; ACM Transactions on Multimedia Computing, Communications, and Applications

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文研究从未分割烹饪视频生成食谱，提出基于transformer的多模态循环方法，训练事件选择器和句子生成器，从DVC输出中选择关键事件并生成基于视觉内容的句子，并引入配料信息，优于现有DVC模型。

摘要：

本文研究从未分割的烹饪视频中生成食谱的任务，该任务要求智能体（1）提取完成菜肴的关键事件，并（2）为提取的事件生成句子。我们的任务类似于密集视频描述（DVC），其目标是全面检测事件并为其生成句子。然而，与DVC不同，在食谱生成中，食谱故事的感知至关重要，模型应以正确的顺序提取适当数量的事件，并基于此生成准确的句子。我们分析了DVC模型的输出，并确认尽管（1）若干事件可作为食谱故事采用，但（2）为这些事件生成的句子并未基于视觉内容。基于此，我们的目标是通过从输出事件中选择预言事件并为它们重新生成句子来获得正确的食谱。为此，我们提出了一种基于transformer的多模态循环方法，训练事件选择器和句子生成器，以从DVC的事件中选择预言事件并为其生成句子。此外，我们通过加入配料来扩展模型，以生成更准确的食谱。实验结果表明，所提出的方法优于最先进的DVC模型。我们还确认，通过以故事感知的方式对食谱进行建模，所提出的模型能够以正确的顺序输出适当数量的事件。

英文摘要：

This paper tackles recipe generation from unsegmented cooking videos, a task that requires agents to (1) extract key events in completing the dish and (2) generate sentences for the extracted events. Our task is similar to dense video captioning (DVC), which aims at detecting events thoroughly and generating sentences for them. However, unlike DVC, in recipe generation, recipe story awareness is crucial, and a model should extract an appropriate number of events in the correct order and generate accurate sentences based on them. We analyze the output of the DVC model and confirm that although (1) several events are adoptable as a recipe story, (2) the generated sentences for such events are not grounded in the visual content. Based on this, we set our goal to obtain correct recipes by selecting oracle events from the output events and re-generating sentences for them. To achieve this, we propose a transformer-based multimodal recurrent approach of training an event selector and sentence generator for selecting oracle events from the DVC's events and generating sentences for them. In addition, we extend the model by including ingredients to generate more accurate recipes. The experimental results show that the proposed method outperforms state-of-the-art DVC models. We also confirm that, by modeling the recipe in a story-aware manner, the proposed model outputs the appropriate number of events in the correct order.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。