VAInpaint：基于LLM驱动模块的零样本视频-音频修复框架
VAInpaint: Zero-Shot Video-Audio inpainting framework with LLMs-driven Module
AI总结：
本文提出VAInpaint框架，通过分割引导视频修复并结合LLM生成的文本查询驱动音频分离，实现零样本音视频联合修复，性能与现有基准相当。
摘要：
针对混合音视频内容的视频与音频修复近年来已成为多媒体编辑中的关键任务。然而，在不影响场景其余部分的情况下精确移除物体及其对应音频仍然是一个重大挑战。为解决这一问题，我们提出了VAInpaint，一种新颖的流程：首先利用分割模型生成掩码并引导视频修复模型移除物体；同时，一个大语言模型（LLM）对场景进行全局分析，而一个区域特定模型提供局部化描述。整体描述和区域描述都将输入到一个LLM中，该LLM对内容进行精炼并将其转化为文本查询，用于我们的文本驱动音频分离模型。我们的音频分离模型在一个定制数据集上进行了微调，该数据集由分割后的MUSIC乐器图像和VGGSound背景组成，以增强其泛化性能。实验表明，我们的方法在音频和视频修复方面均达到了与当前基准相当的性能。
英文摘要：
Video and audio inpainting for mixed audio-visual content has become a crucial task in multimedia editing recently. However, precisely removing an object and its corresponding audio from a video without affecting the rest of the scene remains a significant challenge. To address this, we propose VAInpaint, a novel pipeline that first utilizes a segmentation model to generate masks and guide a video inpainting model in removing objects. At the same time, an LLM then analyzes the scene globally, while a region-specific model provides localized descriptions. Both the overall and regional descriptions will be inputted into an LLM, which will refine the content and turn it into text queries for our text-driven audio separation model. Our audio separation model is fine-tuned on a customized dataset comprising segmented MUSIC instrument images and VGGSound backgrounds to enhance its generalization performance. Experiments show that our method achieves performance comparable to current benchmarks in both audio and video inpainting.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
