Vision-Language Models (VLMs) have become powerful backbones for agents to autonomously operate in digital environments like the web and operating systems. However, these models suffer from inadaptability to fast-changing environments like the web, which can be alleviated by fine-tuning requiring expansive model training and data collection. In this work, we introduce a novel paradigm for enhancing agentic VLM policies at inference without policy retraining. Fundamentally, our approach decouples the VLM's role as a high-capacity action proposer from the final action selection mechanism. We keep the VLM policy frozen and use it to generate a set of candidate actions for a given state. Then, a lightweight, offline-trained Q-function reranks these candidates, and the agent executes the action with the highest estimated value. The main contribution is to apply the Q-function directly during inference for immediate policy improvement, and not offline to relabel data for policy retraining. We demonstrate on the academic WebVoyager benchmark that our method significantly boosts agent success rates, improving a Qwen2.5-VL-7B agent from 38.8% to 55.7% and a proprietary GPT-4.1 agent from 82.4% to 88.8%.
Can Argus Judge Them All? Comparing VLMs Across Domains
Bharati Vidyapeeth(巴哈提大学);Macquarie University(麦考瑞大学);DSEU-Okhla;Vivekananda Institute of Professional Studies(维维kananda专业研究学院);IIIT-Delhi(德里印度理工学院);Center for SDGC(SDGC中心);Stanford University(斯坦福大学)
VLM Can Be a Good Assistant: Enhancing Embodied Visual Tracking with Self-Improving Vision-Language Models
State Key Laboratory of Complex & Critical Software Environment, Beihang University(北京航空航天大学复杂与关键软件环境国家重点实验室);School of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院);City University of Macau(澳门城市大学);Center on Frontiers of Computing Studies, School of Computer Science, Nat’l Eng. Research Center of Visual Technology, Peking University(北京大学计算机学院前沿计算研究中心、国家工程视觉技术研究中心)
The Ebb and Flow of Multimodal Focus: Scheduling Visual Relay Windows for Grounded VLM Reasoning
School of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院);School of Fashion and Textiles, The Hong Kong Polytechnic University(香港理工大学纺织及制衣学院);School of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学人工智能学院)
Multimodal Generative Engine Optimization: Rank Manipulation for Vision-Language Model Rankers
Georgetown University(乔治城大学);University of Southern California(南加州大学);University of Maryland, College Park(马里兰大学学院公园分校);Arizona State University(亚利桑那州立大学)
CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries
University of Macau(澳门大学);Georgia Institute of Technology(佐治亚理工学院);Microsoft Research(微软研究院);Squirrel AI(松鼠AI);Lehigh University(理海大学);William & Mary(威廉玛丽学院)
Multimodal and Multiscale Spatial-Temporal Semantic Search and Recommendation with AI Foundation Models
School of Geographical Sciences and Urban Planning, Arizona State University(地理科学与城市规划学院,亚利桑那州立大学);Alaska Native Tribal Health Consortium(阿拉斯加原住民部落健康联盟);Woodwell Climate Research Center(伍德沃德气候研究中心);School of Computing and Augmented Intelligence, Arizona State University(计算与增强智能学院,亚利桑那州立大学)
Localization Lens for Improving Medical Vision-Language Models
Lahore University of Management Sciences(拉合尔管理科学大学);The University of Western Australia(西澳大学);Information Technology University of the Punjab(旁遮普信息技术大学)
Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding
HKU Musketeers Foundation Institute of Data Science, The University of Hong Kong(香港大学穆斯克特基金会数据科学研究所,香港大学);Department of Electrical and Electronic Engineering, The University of Hong Kong(电气电子工程系,香港大学)