你还记得吗？基于跨模态记忆检索的密集视频字幕生成
Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval
AI总结：
针对仅靠视觉输入完成密集视频字幕生成任务语义不足的问题，提出融合外部记忆先验知识的跨模态记忆检索框架，无需大规模视频预训练即在两个基准数据集上取得良好效果。
摘要：
密集视频字幕生成旨在自动定位并为未修剪视频中的所有事件生成字幕，该研究已受到广泛关注。已有多项研究将密集视频字幕生成设计为事件定位与事件字幕生成的多任务问题，以此考量任务间的关联。然而，由于缺乏语义内容，仅依靠视觉输入完成这两项任务颇具挑战。本研究受人类认知信息处理机制启发，提出一种新型框架来解决该问题。我们的模型利用外部记忆融入先验知识，提出了基于跨模态视频-文本匹配的记忆检索方法。为有效融合检索到的文本特征，设计了带有视觉与文本交叉注意力模块的通用编码器与解码器。我们在ActivityNet Captions和YouCook2数据集上开展了对比实验以验证所提方法的有效性。实验结果表明，在无需基于大型视频数据集进行大规模预训练的情况下，我们的模型取得了颇具前景的性能。
英文摘要：
There has been significant attention to the research on dense video captioning, which aims to automatically localize and caption all events within untrimmed video. Several studies introduce methods by designing dense video captioning as a multitasking problem of event localization and event captioning to consider inter-task relations. However, addressing both tasks using only visual input is challenging due to the lack of semantic content. In this study, we address this by proposing a novel framework inspired by the cognitive information processing of humans. Our model utilizes external memory to incorporate prior knowledge. The memory retrieval method is proposed with cross-modal video-to-text matching. To effectively incorporate retrieved text features, the versatile encoder and the decoder with visual and textual cross-attention modules are designed. Comparative experiments have been conducted to show the effectiveness of the proposed method on ActivityNet Captions and YouCook2 datasets. Experimental results show promising performance of our model without extensive pretraining from a large video dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
