检索即回答：基于冻结大型语言模型的零样本视频问答
Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models
AI总结：
本文提出检索即回答（R2A）框架，通过检索相关文本并利用冻结的大型语言模型实现零样本视频问答，无需跨模态微调，以1.3B参数超越61倍大的Flamingo-80B模型。
摘要：
视频问答（VideoQA）已因近期大型语言模型（LLMs）的扩展而显著进步。其关键思想是将视觉信息转换为语言特征空间，从而充分利用LLMs的能力。现有的VideoQA方法通常采用两种范式：（1）学习跨模态对齐，以及（2）使用现成的字幕模型来描述视觉数据。然而，第一种设计需要在大量额外的多模态数据上进行昂贵的训练，而第二种则进一步受限于有限的领域泛化能力。为解决这些限制，本文提出了一种简单而有效的“检索即回答”（R2A）框架。给定输入视频，R2A首先使用预训练的多模态模型（如CLIP）从通用文本语料库中检索一组语义相似的文本。结合问题和检索到的文本，可以直接使用LLM（如DeBERTa）来生成所需答案。无需跨模态微调，R2A允许所有关键组件（如LLM、检索模型和文本语料库）即插即用。在多个VideoQA基准上的大量实验表明，尽管我们的R2A仅有1.3B参数且未经过微调，但其性能可超越61倍大的Flamingo-80B模型，后者甚至额外在近2.1B多模态数据上进行了训练。
英文摘要：
Video Question Answering (VideoQA) has been significantly advanced from the scaling of recent Large Language Models (LLMs). The key idea is to convert the visual information into the language feature space so that the capacity of LLMs can be fully exploited. Existing VideoQA methods typically take two paradigms: (1) learning cross-modal alignment, and (2) using an off-the-shelf captioning model to describe the visual data. However, the first design needs costly training on many extra multi-modal data, whilst the second is further limited by limited domain generalization. To address these limitations, a simple yet effective Retrieving-to-Answer (R2A) framework is proposed.Given an input video, R2A first retrieves a set of semantically similar texts from a generic text corpus using a pre-trained multi-modal model (e.g., CLIP). With both the question and the retrieved texts, a LLM (e.g., DeBERTa) can be directly used to yield a desired answer. Without the need for cross-modal fine-tuning, R2A allows for all the key components (e.g., LLM, retrieval model, and text corpus) to plug-and-play. Extensive experiments on several VideoQA benchmarks show that despite with 1.3B parameters and no fine-tuning, our R2A can outperform the 61 times larger Flamingo-80B model even additionally trained on nearly 2.1B multi-modal data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
