arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2311.17435cs.CVcs.AI
收藏

MM-Narrator:利用多模态上下文学习为长视频生成解说

MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning

University of Sydney(悉尼大学);Microsoft Azure AI(微软 Azure AI)

Chaoyi Zhang, Kevin Lin, Zhengyuan Yang, Jianfeng Wang, Linjie Li, Chung-Ching Lin, Zicheng Liu, Lijuan Wang

AI总结:

MM-Narrator 利用 GPT-4 的多模态上下文学习与记忆增强生成机制,为超长视频自回归生成故事连贯、角色一致的音频描述,并通过基于复杂度的少样本演示选择提升推理能力,在 MAD-eval 上优于现有方法。

摘要:

我们提出了 MM-Narrator,这是一个利用 GPT-4 和多模态上下文学习来生成音频描述(AD)的新系统。与以往主要侧重于对短视频片段进行下游微调的方法不同,MM-Narrator 擅长以自回归方式为超长视频(甚至超过数小时)生成精确的音频描述。这一能力得益于所提出的记忆增强生成过程,该过程通过高效的“登记-召回”机制,有效利用短期文本上下文和长期视觉记忆。这些上下文记忆汇集了相关的过往信息,包括故事情节和角色身份,确保能够准确追踪并描绘故事连贯且以角色为中心的音频描述。在保持 MM-Narrator 免训练设计的同时,我们进一步提出了一种基于复杂度的演示选择策略,通过少样本多模态上下文学习(MM-ICL)大幅增强其多步推理能力。在 MAD-eval 数据集上的实验结果表明,按照标准评估指标衡量,MM-Narrator 在大多数场景下均一致优于现有的基于微调的方法和基于 LLM 的方法。此外,我们引入了首个面向循环文本生成的基于片段的评估器。该评估器由 GPT-4 驱动,能够在多个可扩展维度上全面推理并评定音频描述生成的表现。

英文摘要:

We present MM-Narrator, a novel system leveraging GPT-4 with multimodal in-context learning for the generation of audio descriptions (AD). Unlike previous methods that primarily focused on downstream fine-tuning with short video clips, MM-Narrator excels in generating precise audio descriptions for videos of extensive lengths, even beyond hours, in an autoregressive manner. This capability is made possible by the proposed memory-augmented generation process, which effectively utilizes both the short-term textual context and long-term visual memory through an efficient register-and-recall mechanism. These contextual memories compile pertinent past information, including storylines and character identities, ensuring an accurate tracking and depicting of story-coherent and character-centric audio descriptions. Maintaining the training-free design of MM-Narrator, we further propose a complexity-based demonstration selection strategy to largely enhance its multi-step reasoning capability via few-shot multimodal in-context learning (MM-ICL). Experimental results on MAD-eval dataset demonstrate that MM-Narrator consistently outperforms both the existing fine-tuning-based approaches and LLM-based approaches in most scenarios, as measured by standard evaluation metrics. Additionally, we introduce the first segment-based evaluator for recurrent text generation. Empowered by GPT-4, this evaluator comprehensively reasons and marks AD generation performance in various extendable dimensions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑