Prevailing joint prediction transformers for Video Highlight Detection and Moment Retrieval (HD/MR) exhibit deficiencies in handling cross-task dynamics, achieving robust video-text alignment, and utilizing effective attention mechanisms, with the potential of Large Language/Vision-Language Models (LLMs/LVLMs) being largely untapped. This paper introduces VideoLights, a novel HD/MR framework addressing these limitations by incorporating: (i) Convolutional Projection and Feature Refinement modules with an alignment loss for enhanced video-text feature congruity; (ii) a Bi-Directional Cross-Modal Fusion network for strongly coupled query-aware representations; (iii) a Uni-directional joint-task feedback mechanism for synergistic task improvement; (iv) hard positive/negative losses for adaptive learning; and (v) the leveraging of LVLMs (e.g., BLIP-2) for superior multimodal feature integration and intelligent pre-training with synthetic data. Comprehensive evaluations on QVHighlights, TVSum, and Charades-STA benchmarks demonstrate that VideoLights significantly surpasses existing baselines, establishing new state-of-the-art performances. Codes and model checkpoints are available at https://github.com/dpaul06/VideoLights .
Re$^3$Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learning
Taobao & Tmall Group of Alibaba(阿里巴巴淘宝和天猫集团);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline
Buzzni AI Lab(Buzzni人工智能实验室);KAIST(韩国科学技术院);Mathpresso Inc.(Mathpresso公司);UC Berkeley(加州大学伯克利分校);Fainders AI(Fainders人工智能公司);Seoul National University(首尔大学)
Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generation
Institute of Computing Technology, CAS(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Pattern Recognition Center, WeChat AI(微信AI模式识别中心)
RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinement
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Wuhan University of Science and Technology(武汉科技大学);National University of Singapore(新加坡国立大学);Northeastern University at Qinhuangdao(东北大学秦皇岛分校);BOSS Zhipin(BOSS直聘)
FAIR-RAG: Faithful Adaptive Iterative Refinement for Retrieval-Augmented Generation
Sharif University of Technology(谢里夫理工大学);Iran University of Science and Technology(伊朗科学技术大学);Noor Avaran Jelvehaye Maanaei Najm Co., Ltd.(诺尔·阿瓦兰·杰拉维耶·马纳埃尼纳姆公司)
Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimization
Zhejiang University(浙江大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);South China University of Technology(华南理工大学);University of British Columbia(不列颠哥伦比亚大学);Fudan University(复旦大学);University of Hong Kong(香港大学)
V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrieval
Tsinghua University(清华大学);University of Central Florida(中央佛罗里达大学);Fudan University(复旦大学);The Australian National University(澳大利亚国立大学);The University of Hong Kong(香港大学);Pengcheng Laboratory(鹏城实验室);Central South University(中南大学)
GraphReAct: Reasoning and Acting for Multi-step Graph Inference
The Chinese University of Hong Kong(香港中文大学);The University of Science and Technology of China(中国科学技术大学);University of Electronic Science and Technology of China(电子科技大学);The University of Tokyo(东京大学);RMIT University(皇家墨尔本理工大学);Singapore Management University(新加坡管理学院)
Multi-modal Fusion and Query Refinement Network for Video Moment Retrieval and Highlight Detection
School of Electronic Science and Engineering, Nanjing University(南京大学电子科学与工程学院);School of Physics and Electronics, Hubei Normal University(湖北师范大学物理与电子科学学院);Gosuncn Chuanglian Technology Co., Ltd.(高新兴创联科技有限公司)
MS-DPPs: Multi-Source Determinantal Point Processes for Contextual Diversity Refinement of Composite Attributes in Text to Image Retrieval
Visual Intelligence Research Laboratories, NEC Corporation(NEC公司视觉智能研究实验室);Graduate School of Information Sciences, Tohoku University(东北大学信息科学研究生院);RIKEN Center for AIP(理化学研究所AIP研究中心)
I2I-PR: Deep Iterative Refinement for Phase Retrieval using Image-to-Image Diffusion Models
Department of Electrical Engineering, Middle East Technical University (METU)(电子工程系,中东技术大学);Department of Applied Mathematics and Computer Science, Technical University of Denmark(应用数学与计算机科学系,丹麦技术大学)
Socializing the Semantic Gap: A Comparative Survey on Image Tag Assignment, Refinement and Retrieval
Renmin University of China(中国人民大学);University of Florence(佛罗伦萨大学);Stanford University(斯坦福大学);University of Amsterdam(阿姆斯特丹大学);Qualcomm Research Netherlands(荷兰高通研究院)
UrbanCross: Enhancing Satellite Image-Text Retrieval with Cross-Domain Adaptation
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Northwestern Polytechnical University(西北工业大学);The Hong Kong University of Science and Technology(香港科技大学)
Cross-Resolution Semantic Transfer for Robust Text-to-Image Retrieval in Low-Resolution Surveillance
School of Computer Science and Technology, Wuhan University of Science and Technology(武汉科技大学计算机科学与技术学院);School of Computer Science, National Engineering Research Center for Multimedia Software, Wuhan University(武汉大学计算机学院,国家多媒体软件工程技术研究中心);Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan University of Science and Technology(湖北省智能信息处理与实时工业系统重点实验室,武汉科技大学)