Key frame selection is essentially a set-level optimization problem: the quality of the selected subset depends on the interactions among frames, rather than the score of any single frame. Existing methods generally exhibit three major limitations. Point-wise methods score each frame independently and ignore inter-frame dependencies. Although the training-free set-level methods explicitly model the inter-frame relationships, their selection criteria are fixed and cannot be adapted through downstream task feedback. Learnable methods can leverage data-driven training; however, they lack an explicit, differentiable set-quality objective and rely on offline-generated supervision signals. To address these limitations, we propose an end-to-end trainable and task-adaptive framework for frame selection. A Chain-of-Thought prompt conditions a Small Language Model (SLM) to extract task-specific latent query vectors, which are combined with multimodal features to enable dynamic, query-aware frame scoring. We further formulate a continuous set-level objective function that jointly accounts for relevance, coverage, and redundancy, enabling differentiable set-level optimization via Gumbel-TopK for selecting optimal frame combinations. Finally, we employ a student-teacher mutual learning strategy, in which the student selector (SLM) and teacher reasoner (MLLM) are trained to align their frame-importance distributions via KL divergence. Combined with cross-entropy loss, this design enables fully end-to-end optimization, eliminating reliance on static pseudo-labels. Experiments across multiple benchmarks, including Video-MME, LongVideoBench, MLVU, and NExT-QA, demonstrate that our method significantly outperforms existing frame-selection approaches.
Routing Before Looking: Query-Adaptive Evidence Acquisition for Long-form Video Understanding
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Tencent(腾讯);Wuhan University(武汉大学);Southeast University(东南大学);National University of Singapore(新加坡国立大学)
Query-focused and Memory-aware Reranker for Long Context Processing
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Pattern Recognition Center, WeChat AI, Tencent(腾讯微信人工智能研究院模式识别中心);East China Normal University(华东师范大学)
HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrieval
School of Software Shandong University Jinan China(软件学院 山东大学 济南 中国);School of Data Science City University of Hong Kong Hong Kong China(数据科学学院 香港城市大学 香港 中国);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));City University of Hong Kong(香港城市大学)
Towards Reliable and Holistic Visual In-Context Learning Prompt Selection
Huazhong University of Science and Technology(华中科技大学);Shanghai Innovation Institute(上海创新研究院);University College London(伦敦大学学院);Tencent Youtu Lab(腾讯优图实验室);The Hong Kong University of Science and Technology(香港理工大学);Fudan University(复旦大学)
Boosting Video-Text Retrieval with Explicit High-Level Semantics
Baidu Inc.(百度公司);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Tianjin University(天津大学);Aberystwyth University(阿伯里斯特威斯大学)
School of Vehicle and Mobility, Tsinghua University(清华大学车辆与出行学院);DiDi Global(滴滴全球公司);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院)