基于程序化程序的零样本视频问答
Zero-Shot Video Question Answering with Procedural Programs
AI总结：
本文提出ProViQ，利用大语言模型生成程序化视频查询程序，通过视觉模块执行子任务，在零样本视频问答及其他视频任务上取得最先进结果。
摘要：
我们提出通过生成简短的程序化程序来回答关于视频的零样本问题，这些程序通过求解一系列视觉子任务来推导出最终答案。我们提出了程序化视频查询（ProViQ），该方法使用大型语言模型根据输入问题和提示中的视觉模块API生成此类程序，然后执行这些程序以获得输出。近期类似的程序化方法已在图像问答中取得成功，但视频仍然具有挑战性：我们为ProViQ提供了用于视频理解的模块，使其能够泛化到各种视频。该代码生成框架还使ProViQ除了问答之外还能执行其他视频任务，例如多目标跟踪或基础视频编辑。ProViQ在多种基准测试上取得了最先进的结果，在短、长、开放式和多模态视频问答数据集上最高提升了25%。我们的项目页面位于https://rccchoudhury.github.io/proviq2023。
英文摘要：
We propose to answer zero-shot questions about videos by generating short procedural programs that derive a final answer from solving a sequence of visual subtasks. We present Procedural Video Querying (ProViQ), which uses a large language model to generate such programs from an input question and an API of visual modules in the prompt, then executes them to obtain the output. Recent similar procedural approaches have proven successful for image question answering, but videos remain challenging: we provide ProViQ with modules intended for video understanding, allowing it to generalize to a wide variety of videos. This code generation framework additionally enables ProViQ to perform other video tasks in addition to question answering, such as multi-object tracking or basic video editing. ProViQ achieves state-of-the-art results on a diverse range of benchmarks, with improvements of up to 25% on short, long, open-ended, and multimodal video question-answering datasets. Our project page is at https://rccchoudhury.github.io/proviq2023.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
