arXivDaily arXiv每日学术速递 周一至周五更新
​通知:联邦节假日调休:哥伦布日 / 原住民日(Columbus Day / Indigenous Peoples' Day),今日(13日)无更新。
arXiv 2503.12077cs.CVcs.AI

V-Stylist:通过MLLM智能体的协作与反思实现视频风格化

V-Stylist: Video Stylization via Collaboration and Reflection of MLLM Agents

Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学)

Zhengrong Yue, Shaobin Zhuang, Kunchang Li, Yanbo Ding, Yali Wang

AI总结:

本文提出多智能体视频风格化系统V-Stylist,通过视频解析、风格树搜索和多轮自我反思协作处理复杂转场与开放风格描述,并构建TVSBench,性能达到SOTA。

摘要:

尽管视频风格化近期取得了进展,但大多数现有方法在基于用户查询的开放风格描述来渲染包含复杂转场的任意视频时仍面临困难。为填补这一空白,我们提出了一种用于视频风格化的通用多智能体系统V-Stylist,该系统采用多模态大语言模型的新型协作与反思范式。具体而言,V-Stylist是一个包含三个关键角色的系统化工作流:(1)Video Parser将输入视频分解为若干镜头,并生成这些镜头关键内容的文本提示。通过简洁的视频到镜头提示范式,它使V-Stylist能够有效处理具有复杂转场的视频。(2)Style Parser识别用户查询中的风格,并从风格树中逐步搜索匹配的风格模型。通过鲁棒的思维树搜索范式,它使V-Stylist能够精确明确开放用户查询中模糊的风格偏好。(3)Style Artist利用匹配的模型将所有视频镜头渲染为所需风格。通过新颖的多轮自我反思范式,它使V-Stylist能够根据风格要求自适应地调整细节控制。凭借这种模仿人类专业人员的独特设计,V-Stylist在有效且自动的视频风格化所面临的主要挑战上实现了重大突破。此外,我们进一步构建了一个新的基准Text-driven Video Stylization Benchmark(TVSBench),填补了在开放用户查询下评估复杂视频风格化的空白。大量实验表明,V-Stylist达到了最先进水平,例如,V-Stylist在总体平均指标上分别超过FRESCO和ControlVideo 6.05%和4.51%,标志着视频风格化的显著进步。

英文摘要:

Despite the recent advancement in video stylization, most existing methods struggle to render any video with complex transitions, based on an open style description of user query. To fill this gap, we introduce a generic multi-agent system for video stylization, V-Stylist, by a novel collaboration and reflection paradigm of multi-modal large language models. Specifically, our V-Stylist is a systematical workflow with three key roles: (1) Video Parser decomposes the input video into a number of shots and generates their text prompts of key shot content. Via a concise video-to-shot prompting paradigm, it allows our V-Stylist to effectively handle videos with complex transitions. (2) Style Parser identifies the style in the user query and progressively search the matched style model from a style tree. Via a robust tree-of-thought searching paradigm, it allows our V-Stylist to precisely specify vague style preference in the open user query. (3) Style Artist leverages the matched model to render all the video shots into the required style. Via a novel multi-round self-reflection paradigm, it allows our V-Stylist to adaptively adjust detail control, according to the style requirement. With such a distinct design of mimicking human professionals, our V-Stylist achieves a major breakthrough over the primary challenges for effective and automatic video stylization. Moreover,we further construct a new benchmark Text-driven Video Stylization Benchmark (TVSBench), which fills the gap to assess stylization of complex videos on open user queries. Extensive experiments show that, V-Stylist achieves the state-of-the-art, e.g.,V-Stylist surpasses FRESCO and ControlVideo by 6.05% and 4.51% respectively in overall average metrics, marking a significant advance in video stylization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑