arXivDaily arXiv每日学术速递 周一至周五更新
​通知:联邦节假日调休:哥伦布日 / 原住民日(Columbus Day / Indigenous Peoples' Day),今日(13日)无更新。
arXiv 2604.17475cs.AIcs.CLcs.LG

觉醒失明:为基于视觉感知的代理轨迹进行无监督的冷启动优化

Waking Up Blind: Cold-Start Optimization of Supervision-Free Agentic Trajectories for Grounded Visual Perception

Indian Institute of Technology Delhi(印度理工学院德里);Indian Institute of Technology Abu Dhabi(印度理工学院阿布扎比);Microsoft Research(微软研究院)

Ashutosh Bajpai, Tamal Majumder, Akshay Nambi, Tanmoy Chakraborty

AI总结:

本文提出SPECTRA框架,通过冷启动强化学习提升小视觉语言模型的视觉鲁棒性和工具协调能力,无需监督即可提高任务准确性和工具效率。

摘要:

本文提出SPECTRA框架,通过冷启动强化学习提升小视觉语言模型的视觉鲁棒性和工具协调能力,无需监督即可提高任务准确性和工具效率。

英文摘要:

Small Vision-Language Models (SVLMs) are efficient task controllers but often suffer from visual brittleness and poor tool orchestration. They typically require expensive supervised trajectory tuning to mitigate these deficits. In this work, we propose Self-supervised Perception Enabled by Cascaded Tool Rollout Alignment (SPECTRA), a supervision-free framework that bootstraps agentic capabilities via Coldstart Reinforcement Learning for SVLMs. SPECTRA enforces Soft Structured Multi-turn Rollouts, a topological constraint that directs agents to explicitly sequence tool derived evidence before synthesis, effectively grounding reasoning in visual observations. We employ a multi-objective reward signal that simultaneously maximizes task correctness, rollout structure, and tool utility, enabling agent to self-discover robust behaviors without human preference labels. We further introduce Tool Instrumental Utility (TIU), a novel metric to quantify tool efficacy in the absence of ground truth. Extensive evaluations across composite and out-of-distribution (MMMU-Pro) benchmarks demonstrate that SPECTRA boosts agentic trajectories, improving task accuracy by up to 5% and tool efficiency by 9%, enabling more efficient multimodal agents that learn effectively from environmental interaction alone.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑