CPPO: 面向VLM智能体的对比感知策略优化
CPPO: Contrastive Perception Policy Optimization for VLM Agents
AI总结：
提出一种自监督的对比感知策略优化方法CPPO，通过对比感知损失增强视觉语言模型的视觉基础能力，无需额外模型或标注，在感知关键任务中优于现有方法。
摘要：
我们引入了CPPO，一种用于微调视觉语言模型（VLM）的对比感知策略优化方法。可靠的感知是基于VLM的智能体在开放环境中推理和行动的核心要求：错误的视觉基础直接导致错误的行为、幻觉工具调用和不安全的决策。虽然强化学习（RL）显著提升了语言模型的推理能力，但将这些进展扩展到多模态智能体需要同时改进感知和推理。先前的工作主要通过显式感知奖励来解决这一挑战，这通常需要额外的LLM评判器、真实标注或强制将感知与推理分离。CPPO通过扩展RL目标，引入对比感知损失（CPL），以自监督方式解决了这一限制，为视觉基础提供了直接的学习信号。对比目标鼓励模型对输入的视觉信息更加敏感。为了有效应用这一信号，CPPO利用在扰动图像下模型输出分布中的熵移机制识别感知令牌，并在训练期间选择性地对这些令牌应用对比损失。实验表明，CPPO在避免额外模型的同时超越了先前方法，使训练更加高效和可扩展，并产生了更适合感知关键智能体任务的策略。
英文摘要：
We introduce CPPO, a Contrastive Perception Policy Optimization method for finetuning vision--language models (VLMs). Reliable perception is a core requirement for VLM-based agents that must reason and act in open-ended environments: faulty visual grounding cascades directly into faulty actions, hallucinated tool calls, and unsafe decisions. While reinforcement learning (RL) has significantly improved reasoning in language models, extending these advances to multimodal agents requires improving both perception and reasoning. Prior works address this challenge mainly through explicit perception rewards, which often require extra LLM judges, ground-truth annotations, or forced separation of perception from reasoning. CPPO addresses this limitation in a self-supervised manner by extending the RL objective with a Contrastive Perception Loss (CPL) that provides a direct learning signal for visual grounding. The contrastive objective encourages the model to become more sensitive to input visual information. To apply this signal effectively, CPPO identifies perception tokens using an entropy-shift mechanism in the model's output distributions under perturbed images and applies the contrastive loss selectively to those tokens during training. Experiments show that CPPO surpasses prior methods while avoiding extra models, making training more efficient and scalable, and yielding policies that are better suited to perception-critical agentic tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
