arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.18850cs.CV
收藏

HORNet:基于任务引导的帧选择用于视觉语言模型的视频问答

HORNet: Task-Guided Frame Selection for Video Question Answering with Vision-Language Models

Northeastern University(东北大学)

Xiangyu Bai, Bishoy Galoaa, Sarah Ostadabbas

AI总结:

HORNet通过轻量级策略优化,显著减少视频输入帧数和VLM处理时间,同时提升问答质量与时间推理任务表现。

摘要:

HORNet通过轻量级策略优化,显著减少视频输入帧数和VLM处理时间,同时提升问答质量与时间推理任务表现。

英文摘要:

Video question answering (VQA) with vision-language models (VLMs) depends critically on which frames are selected from the input video, yet most systems rely on uniform or heuristic sampling that cannot be optimized for downstream answering quality. We introduce \textbf{HORNet}, a lightweight frame selection policy trained with Group Relative Policy Optimization (GRPO) to learn which frames a frozen VLM needs to answer questions correctly. With fewer than 1M trainable parameters, HORNet reduces input frames by up to 99\% and VLM processing time by up to 93\%, while improving answer quality on short-form benchmarks (+1.7\% F1 on MSVD-QA) and achieving strong performance on temporal reasoning tasks (+7.3 points over uniform sampling on NExT-QA). We formalize this as Select Any Frames (SAF), a task that decouples visual input curation from VLM reasoning, and show that GRPO-trained selection generalizes better out-of-distribution than supervised and PPO alternatives. HORNet's policy further transfers across VLM answerers without retraining, yielding an additional 8.5\% relative gain when paired with a stronger model. Evaluated across six benchmarks spanning 341,877 QA pairs and 114.2 hours of video, our results demonstrate that optimizing \emph{what} a VLM sees is a practical and complementary alternative to optimizing what it generates while improving efficiency. Code is available at https://github.com/ostadabbas/HORNet.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑