通过通用关键帧提取桥接VideoQA和视频引导的智能体任务
Bridging VideoQA and Video-Guided Agentic Tasks via Generalized Keyframe Extraction
AI总结：
提出VG-GUIBench基准和TASKER关键帧提取算法，联合考虑任务相关性和场景动态，在VideoQA和视频引导的GUI任务上提升性能。
摘要：
视频理解是多模态智能的基本能力，最近的多模态大语言模型（MLLMs）在视频问答（VideoQA）基准上取得了显著性能。然而，现有基准主要评估模型是否能感知浅层视觉线索，很少检验MLLMs能否从视频教程中学习更深层的知识或程序性技能，并将其泛化到下游的长周期智能体任务。为填补这一空白，我们引入VG-GUIBench（视频引导的GUI基准），这是一个新基准，旨在评估基于MLLM的GUI智能体能否跟随视频教程完成相应的GUI交互任务。此外，我们观察到模型在VideoQA和视频引导的智能体任务上的性能都关键依赖于有效的关键帧提取。基于这一观察，我们提出TASKER（任务驱动和场景感知的关键帧搜索器），一种联合考虑任务相关性和场景动态以识别信息帧的关键帧提取算法。实验结果表明，TASKER在VideoQA和视频引导的智能体任务基准上都取得了显著的性能提升，在EgoSchema全集上比最佳基线高出2.0%，在NExT-QA数据集上高出1.8%。这些结果进一步突显了通用关键帧提取方法在视频理解任务中的潜力。我们的代码和数据可在https://github.com/VG-GUI-TASKER/VG-GUI-TASKER获取。
英文摘要：
Video understanding is a fundamental capability for multimodal intelligence, and recent Multimodal Large Language Models (MLLMs) have achieved remarkable performance on Video Question Answering (VideoQA) benchmarks. However, existing benchmarks primarily evaluate whether models can perceive shallow visual cues, while rarely examining whether MLLMs can learn deeper knowledge or procedural skills from video tutorials and generalize them to downstream long-horizon agentic tasks. To address this gap, we introduce VG-GUIBench (Video-Guided GUI Benchmark), a new benchmark designed to evaluate whether MLLM-based GUI agents can follow video tutorials to complete corresponding GUI interactive tasks. Furthermore, we observe that the performance of models on both VideoQA and video-guided agentic tasks critically depends on effective keyframe extraction. Based on this observation, we propose TASKER (Task-driven And Scene-aware Keyframe searchER), a keyframe extraction algorithm that jointly considers task relevance and scene dynamics to identify informative frames. Experimental results demonstrate that TASKER achieves significant performance improvements on both VideoQA and video-guided agentic task benchmarks, outperforming the best baseline by 2.0% on the EgoSchema fullset and 1.8% on the NExT-QA dataset, respectively. These results further highlight the potential of generalized keyframe extraction methods for video understanding tasks. Our code and data are available at https://github.com/VG-GUI-TASKER/VG-GUI-TASKER.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
