arXiv 2606.147832026-06-16cs.CVcs.CR

视觉编码器作为隐私边界：无编码器视觉-语言模型中的视觉令牌侧信道

The Vision Encoder as a Privacy Boundary: Visual-Token Side Channels in Encoder-Free Vision-Language Models

School of Engineering, Institute of Science Tokyo（东京科学大学工学院）；College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院）；Department of Electrical and Computer Engineering, National University of Singapore（新加坡国立大学电气与计算机工程系）

Chenyu Zhou, Qiliang Jiang, Shuning Wu, Xu Zhou

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

研究无编码器视觉-语言模型中视觉令牌侧信道导致的隐私泄露问题，通过解码器攻击从中间视觉令牌恢复图像和文本，发现空间采样保真度是关键因素，并指出KV缓存也存在泄露风险。

摘要：

视觉编码器将图像像素压缩为语义嵌入，通过保留语义内容同时衰减精确文本恢复所需的像素局部细节，隐式地充当隐私边界。无编码器视觉-语言模型（VLM）通过将图像块直接路由到语言模型令牌流中移除了这一边界，从而暴露了一个架构上的隐私攻击面：中间视觉令牌成为输出前的侧信道。在令牌访问攻击者下，解码器从两个无编码器VLM（Gemma4和Fuyu）中反转视觉令牌流，恢复可识别的图像结构和可读的保留访问码，而匹配的基于编码器的控制模型仅能定位目标区域但无法恢复精确字符串。模型内消融实验表明，操作因素是视觉令牌网格的空间采样保真度，尤其是字符方向采样密度，而非令牌或值的数量。泄露不仅限于导出的令牌：Gemma4第0层键值缓存张量可直接反转，将侧信道置于生产服务栈通常为解码效率而持久化的KV缓存中。该攻击在杂乱场景、真实文档退化以及零样本迁移到公共文档图像中依然有效，并抵抗加性噪声和量化等值级防御。因此，有效的缓解措施必须降低空间采样，使得移除视觉编码器成为VLM部署中的一级隐私决策。

英文摘要：

A vision encoder compresses image pixels into semantic embeddings, implicitly acting as a privacy boundary by preserving semantic content while attenuating pixel-local detail required for exact text recovery. Encoder-free vision-language models (VLMs) remove this boundary by routing image patches directly into the language-model token stream, thereby exposing an architectural privacy attack surface: intermediate visual tokens become a pre-output side channel. Under a token-access adversary, decoders invert visual-token streams from two encoder-free VLMs, Gemma4 and Fuyu, recovering recognizable image structure and readable held-out access codes, whereas matched encoder-based controls localize target regions but recover no exact strings. Within-model ablations show that the operative factor is spatial sampling fidelity of the visual-token grid, especially character-direction sampling density, rather than token or value count. The leakage is not limited to exported tokens: Gemma4 layer-0 key-value cache tensors are directly invertible, placing the side channel within KV caches commonly persisted by production serving stacks for decoding efficiency. The attack survives clutter, realistic document degradation, and zero-shot transfer to public document images, and it resists value-level defenses such as additive noise and quantization. Effective mitigation must therefore reduce spatial sampling, making removal of the vision encoder a first-class privacy decision in VLM deployment.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。