Pixelis:在像素中推理,从看见到行动
Pixelis: Reasoning in Pixels, from Seeing to Acting
AI总结:
Pixelis通过在像素空间中直接操作图像和视频,结合执行操作和学习后果,提升视觉智能的通用性和物理基础,实现基于行动的多模态感知。
摘要:
大多数视觉-语言系统都是静态观察者:它们描述像素,不行动,无法在变化中安全改进。这种被动性限制了可推广的、基于物理的视觉智能。通过行动而非静态描述进行学习是超越受控数据的关键。我们提出了Pixelis,一个在像素空间中操作的智能体,通过一组紧凑的可执行操作(缩放/裁剪、分割、跟踪、OCR、时间定位)直接在图像和视频上操作,并通过其后果学习。Pixelis在三个阶段训练:(1)监督微调学习链式思维-行动轨迹中的像素工具语法,使用掩码模仿损失,强调操作/参数标记和辅助头以稳定像素基础论点;(2)好奇心-一致性奖励微调优化预测误差好奇心与相邻步骤一致性及温和效率先验的双驱动目标,在KL锚定下产生短、有效、结构化的工具链;(3)像素测试时间强化学习通过检索邻居、对完整轨迹投票而非答案、更新向短、高保真范例,同时用KL到EMA安全控制限制漂移。在六个公开图像和视频基准上,Pixelis实现了持续改进:平均相对增益为8B基线的+4.08%(在VSI-Bench达到+6.03%),计算为(我们的-基线)/基线,同时产生更短、可审计的工具链并保持测试时间学习中的KL值。在像素中行动而非抽象标记,将多模态感知扎根于物理世界,连接视觉推理与可行动结果,并实现无外部反馈的具身适应。
英文摘要:
Most vision-language systems are static observers: they describe pixels, do not act, and cannot safely improve under shift. This passivity limits generalizable, physically grounded visual intelligence. Learning through action, not static description, is essential beyond curated data. We present Pixelis, a pixel-space agent that operates directly on images and videos via a compact set of executable operations (zoom/crop, segment, track, OCR, temporal localization) and learns from its consequences. Pixelis trains in three phases: (1) Supervised Fine-Tuning learns a pixel-tool grammar from Chain-of-Thought-Action traces with a masked imitation loss that upweights operation/argument tokens and auxiliary heads to stabilize pixel-grounded arguments; (2) Curiosity-Coherence Reward Fine-Tuning optimizes a dual-drive objective marrying prediction-error curiosity with adjacent-step coherence and a mild efficiency prior under a KL anchor, yielding short, valid, structured toolchains; (3) Pixel Test-Time RL performs label-free adaptation by retrieving neighbors, voting over complete trajectories rather than answers, and updating toward short, high-fidelity exemplars while constraining drift with a KL-to-EMA safety control. Across six public image and video benchmarks, Pixelis yields consistent improvements: the average relative gain is +4.08% over the same 8B baseline (peaking at +6.03% on VSI-Bench), computed as (ours-baseline)/baseline, while producing shorter, auditable toolchains and maintaining in-corridor KL during test-time learning. Acting within pixels, rather than abstract tokens, grounds multimodal perception in the physical world, linking visual reasoning with actionable outcomes, and enables embodied adaptation without external feedback.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。