arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.06445cs.CVcs.AI
收藏

代理分析:作为条件编码器的视觉语言模型中的定位信号

Analysis-by-Proxy: Localization Signals in VLMs Operating as Condition Encoders

Tel Aviv University, Tel Aviv, Israel(特拉维夫大学);Google DeepMind(谷歌DeepMind)

Yoav Baron, Sara Dorfman, Roni Paiss, Daniel Cohen-Or, Or Patashnik

AI总结:

研究视觉语言模型(VLMs)作条件编码器时编辑管道定位性能差的问题,引入代理分析框架,通过训练代理模型分析VLM中间表示来揭示编码定位信息的表示,发现现有条件提取策略缺陷,为条件架构设计提供新思路。

摘要:

视觉语言模型(VLMs)因其卓越的多模态推理能力,越来越多地被用作基于扩散的图像编辑的条件主干。虽然独立的VLMs显示出强大的定位能力,但编辑管道在保持这种准确性方面常常遇到困难,特别是在复杂的多实体场景中。在这项工作中,我们研究了这种性能差距,假设它源于将VLM视为条件编码器。在这个角色中,模型被限制为单次前向传递,阻止了其经过优化的自回归生成过程,从而无法充分发挥其能力。为了研究当VLM用作条件编码器时这种空间理解是否仍然存在,我们引入了代理分析。在这个框架中,我们使用辅助定位任务在VLM的中间表示上训练一个轻量级、可解释的代理模型。通过这个代理分析VLM,我们发现了编码定位信息的特定VLM表示。我们的发现揭示了VLM条件编码器中空间知识的表示方式与当前编辑管道提取方式之间的根本不匹配。我们发现,在单次传递约束下,定位信号不能可靠地传播到通常用于条件设定的预定义层配置。相反,这个关键信号隐藏在中间表示中,其位置因输入提示而异。使用我们引入的代理分析框架,我们揭示了编辑管道中现有条件提取策略的根本缺陷,为更有原则的条件架构设计打开了大门。

英文摘要:

Vision-Language Models (VLMs) are increasingly utilized as the conditioning backbone for diffusion-based image editing due to their remarkable multimodal reasoning capabilities. While standalone VLMs demonstrate strong localization capabilities, editing pipelines frequently struggle to maintain this accuracy, particularly in complex, multi-entity scenes. In this work, we investigate this performance gap, hypothesizing that it stems from treating the VLM as a condition encoder. In this role, the model is restricted to a single forward pass, preventing the autoregressive generation process for which it was optimized, thereby failing to fully expose its capabilities. To investigate whether this spatial understanding persists when the VLM is used as a condition encoder, we introduce Analysis-by-Proxy. In this framework, we train a lightweight, interpretable proxy model on the VLM's intermediate representations using an auxiliary localization task. By analyzing the VLM through this proxy, we uncover the specific VLM representations that encode localization information. Our findings expose a fundamental mismatch between how spatial knowledge is represented within a VLM condition encoder and how it is extracted by current editing pipelines. We reveal that under single-pass constraints, the localization signal does not reliably propagate to the predefined layer configurations commonly used for conditioning. Instead, this crucial signal remains hidden within intermediate representations, at locations that vary depending on the input prompt. Using our introduced Analysis-by-Proxy framework, we reveal the fundamental failures of existing condition extraction strategies in editing pipelines, opening the door to more principled design of conditioning architectures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑