arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.14938cs.CLcs.AI
收藏

面向大型视频库的检索增强生成研究

Towards Retrieval Augmented Generation over Large Video Libraries

Moments Lab Research(Moments Lab 研究院)

Yannis Tevissen, Khalil Guetari, Frédéric Petitpont

AI总结:

本文提出视频库问答(VLQA)任务,构建适配视频库的RAG互操作架构,用LLMs生成查询检索带语音和视觉元数据的视频片段,再结合查询生成带时间戳的回答,助力多媒体检索与AI视频创作。

摘要:

视频内容创作者需要高效工具来二次利用内容,这类任务通常需要复杂的手动或自动搜索。从大型视频库中制作新视频仍是一项挑战。本文提出了视频库问答(Video Library Question Answering,VLQA)任务,通过一套可互操作的架构将检索增强生成(Retrieval Augmented Generation,RAG)应用于视频库。我们提出的系统利用大语言模型(large language models,LLMs)生成搜索查询,检索经语音和视觉元数据索引的相关视频片段。随后答案生成模块将用户查询与这些元数据相结合,生成带有具体视频时间戳的响应。该方法在多媒体内容检索、AI辅助视频内容创作领域展现出应用前景。

英文摘要:

Video content creators need efficient tools to repurpose content, a task that often requires complex manual or automated searches. Crafting a new video from large video libraries remains a challenge. In this paper we introduce the task of Video Library Question Answering (VLQA) through an interoperable architecture that applies Retrieval Augmented Generation (RAG) to video libraries. We propose a system that uses large language models (LLMs) to generate search queries, retrieving relevant video moments indexed by speech and visual metadata. An answer generation module then integrates user queries with this metadata to produce responses with specific video timestamps. This approach shows promise in multimedia content retrieval, and AI-assisted video content creation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑