利用领域特定细粒度启发式提示视频-语言基础模型进行视频问答
Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering
AI总结：
提出HeurVidQA框架，通过领域特定实体-动作启发式提示优化预训练视频-语言基础模型，显著提升视频问答的推理准确性和上下文感知能力。
摘要：
视频问答（VideoQA）代表了视频理解与语言处理之间的关键交叉点，既需要判别性的单模态理解，又需要复杂的跨模态交互以实现准确推理。尽管多模态预训练模型和视频-语言基础模型取得了进展，但由于其泛化的预训练目标，这些系统在处理领域特定的VideoQA时往往表现不佳。解决这一差距需要弥合广泛跨模态知识与VideoQA任务特定推理需求之间的鸿沟。为此，我们提出了HeurVidQA框架，该框架利用领域特定的实体-动作启发式来优化预训练的视频-语言基础模型。我们的方法将这些模型视为隐式知识引擎，采用领域特定的实体-动作提示器引导模型关注增强推理的精确线索。通过提供细粒度启发式，我们提升了模型识别和解释关键实体与动作的能力，从而增强其推理能力。在多个VideoQA数据集上的广泛评估表明，我们的方法显著优于现有模型，强调了将领域特定知识整合到视频-语言模型中以实现更准确和上下文感知的VideoQA的重要性。
英文摘要：
Video Question Answering (VideoQA) represents a crucial intersection between video understanding and language processing, requiring both discriminative unimodal comprehension and sophisticated cross-modal interaction for accurate inference. Despite advancements in multi-modal pre-trained models and video-language foundation models, these systems often struggle with domain-specific VideoQA due to their generalized pre-training objectives. Addressing this gap necessitates bridging the divide between broad cross-modal knowledge and the specific inference demands of VideoQA tasks. To this end, we introduce HeurVidQA, a framework that leverages domain-specific entity-action heuristics to refine pre-trained video-language foundation models. Our approach treats these models as implicit knowledge engines, employing domain-specific entity-action prompters to direct the model's focus toward precise cues that enhance reasoning. By delivering fine-grained heuristics, we improve the model's ability to identify and interpret key entities and actions, thereby enhancing its reasoning capabilities. Extensive evaluations across multiple VideoQA datasets demonstrate that our method significantly outperforms existing models, underscoring the importance of integrating domain-specific knowledge into video-language models for more accurate and context-aware VideoQA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
