CogStream: 基于上下文的流视频问答
CogStream: Context-guided Streaming Video Question Answering
AI总结：
本文提出CogStream任务，通过视觉流压缩和历史对话检索，解决流视频中关键信息识别与问答问题。
摘要：
尽管视频大语言模型（Vid-LLMs）在多模态理解方面取得了进步，但流视频推理仍面临挑战，因为其依赖于上下文信息。现有方法将所有可用的历史上下文信息输入Vid-LLMs，导致视觉数据处理的计算负担显著增加。此外，无关上下文的引入会干扰模型对关键细节的识别。本文引入了一个具有挑战性的任务，称为基于上下文的流视频推理（CogStream），它模拟了现实世界中的流视频场景，要求模型识别最相关的歷史上下文信息以推断当前流的问题答案。为了支持CogStream，我们提出了一个密集标注的数据集，包含大量且层次分明的问题-答案对，由半自动流程生成。此外，我们还提出了CogReasoner作为基线模型。该模型通过利用视觉流压缩和历史对话检索有效解决了这一任务。广泛的实验证明了该方法的有效性。
英文摘要：
Despite advancements in Video Large Language Models (Vid-LLMs) improving multimodal understanding, challenges persist in streaming video reasoning due to its reliance on contextual information. Existing paradigms feed all available historical contextual information into Vid-LLMs, resulting in a significant computational burden for visual data processing. Furthermore, the inclusion of irrelevant context distracts models from key details. This paper introduces a challenging task called Context-guided Streaming Video Reasoning (CogStream), which simulates real-world streaming video scenarios, requiring models to identify the most relevant historical contextual information to deduce answers for questions about the current stream. To support CogStream, we present a densely annotated dataset featuring extensive and hierarchical question-answer pairs, generated by a semi-automatic pipeline. Additionally, we present CogReasoner as a baseline model. It effectively tackles this task by leveraging visual stream compression and historical dialogue retrieval. Extensive experiments prove the effectiveness of this method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
