arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.09230cs.CVcs.AI
收藏

基础模型与自适应特征选择:一种面向视频问答的协同方法

Foundation Models and Adaptive Feature Selection: A Synergistic Approach to Video Question Answering

Indian Institute of Technology Bombay(印度理工学院孟买分校);LNMIIT(拉克希米·尼瓦思技术与管理学院);Fractal AI Research(分形人工智能研究院);Univesity of Houston(休斯顿大学)

Sai Bhargav Rongali, Mohamad Hassan N C, Ankit Jha, Neha Bhargava, Saurabh Prasad, Biplab Banerjee

AI总结:

针对视频问答中问题与视频多模态信息整合不足的问题,提出LGQAVE框架,通过交叉注意力选帧、问题感知动态图Transformer及多粒度嵌入融合,结合miniGPT等模型,在多个基准上的多项选择与开放式问答任务性能显著优于现有方法。

摘要:

本文致力于解决视频问答(VideoQA)这一复杂挑战。尽管现有研究已取得显著进展,但当前方法仍无法有效将问题与视频帧、语义对象级抽象相整合,以生成具备问题感知能力的视频表征。\n我们提出了局部-全局问题感知视频嵌入(Local-Global Question Aware Video Embedding,LGQAVE)框架,该框架包含三项核心创新,旨在更好地融合多模态知识,并突出与特定问题相关的语义视觉概念。LGQAVE 摒弃了传统的临时帧采样方式,采用交叉注意力机制精准识别与问题最相关的视频帧;它通过独立的图结构捕捉这些帧内的对象动态,并借助 miniGPT 模型将其锚定在问题语义中。这些图由问题感知动态图Transformer(question-aware dynamic graph transformer,Q-DGT)处理,对输出进行优化以生成精细的全局与局部视频表征。额外的交叉注意力模块将这些局部与全局嵌入相融合,生成最终的视频嵌入,供语言模型用于生成答案。\n在多个基准上的广泛评估表明,LGQAVE 在生成准确的多项选择答案和开放式答案方面,性能显著优于现有模型。

英文摘要:

This paper tackles the intricate challenge of video question-answering (VideoQA). Despite notable progress, current methods fall short of effectively integrating questions with video frames and semantic object-level abstractions to create question-aware video representations. We introduce Local-Global Question Aware Video Embedding (LGQAVE), which incorporates three major innovations to integrate multi-modal knowledge better and emphasize semantic visual concepts relevant to specific questions. LGQAVE moves beyond traditional ad-hoc frame sampling by utilizing a cross-attention mechanism that precisely identifies the most relevant frames concerning the questions. It captures the dynamics of objects within these frames using distinct graphs, grounding them in question semantics with the miniGPT model. These graphs are processed by a question-aware dynamic graph transformer (Q-DGT), which refines the outputs to develop nuanced global and local video representations. An additional cross-attention module integrates these local and global embeddings to generate the final video embeddings, which a language model uses to generate answers. Extensive evaluations across multiple benchmarks demonstrate that LGQAVE significantly outperforms existing models in delivering accurate multi-choice and open-ended answers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑