arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.12271cs.CV
收藏

超越文本提示:视觉到视觉生成作为统一范式

Beyond Text Prompts: Visual-to-Visual Generation as A Unified Paradigm

City University of Hong Kong(香港城市大学);City University of Hong Kong (Dongguan)(香港城市大学(东莞));The Hong Kong University of Science and Technology(香港科学与技术大学);The Chinese University of Hong Kong(香港中文大学);Celia Research HK(Celia研究香港);Great Bay University(大湾大学);Lingnan University(岭南大学)

Yaofang Liu, Kangning Cui, Meng Chu, Zhaoqing Li, Suiyun Zhang, Jean-Michel Morel, Xiaodong Cun, Haoxuan Che, Rui Liu, Raymond H. Chan

AI总结:

提出视觉到视觉(V2V)生成范式及无需训练的V2V-Zero框架,通过利用视觉页面隐藏状态替代文本条件,在多个任务上达到或接近优化后的文本到图像性能。

摘要:

人类通常通过视觉制品(如排版表、草图、参考图像和标注场景)来指定和创作。然而,现代视觉生成器仍然要求用户将这种意图序列化为文本,这一瓶颈压缩了空间结构、精确外观和字形形状等信号。我们提出 extbf{\emph{视觉到视觉}(V2V)}生成,其中用户使用视觉规范页面(而非文本提示)来条件化生成模型。该页面不是编辑目标,而是指定所需输出的视觉文档。我们引入 extbf{V2V-Zero},一个无需训练的框架,通过用从视觉页面提取的最终层隐藏状态替换纯文本条件,在现有的视觉语言模型(VLM)条件化生成器中暴露此接口,利用了冻结的VLM已将文本和图像映射到生成器条件空间的事实。在GenEval上,V2V-Zero使用冻结的Qwen-Image骨干网络达到0.85,接近其优化后的文本到图像性能而无需微调。为评估更广泛的V2V空间,我们引入 extbf{Simple-V2V Bench},涵盖七个视觉条件化任务和七个模型,包括GPT Image 2、Nano Banana 2、Seedream 5.0 Lite、开源权重基线和视频扩展。V2V-Zero得分为32.7/100,优于评估的开源图像基线,并揭示了清晰的能力层次:属性绑定强,内容生成不可靠,结构控制即使对商业系统也困难。HunyuanVideo-1.5扩展得分为20.2/100,表明该接口可迁移到图像之外。机制分析显示默认推理路径主要通过视觉路由,95.0%的条件化token注意力集中在视觉页面隐藏状态上。

英文摘要:

Humans often specify and create through visual artifacts: typography sheets, sketches, reference images, and annotated scenes. Yet modern visual generators still ask users to serialize this intent into text, a bottleneck that compresses signals like spatial structure, exact appearance, and glyph shape. We propose \textbf{\emph{visual-to-visual} (V2V)} generation, in which the user conditions a generative model with a visual specification page rather than a text prompt. The page is not an edit target, but a visual document that specifies the desired output. We introduce \textbf{V2V-Zero}, a training-free framework that exposes this interface in existing vision-language model (VLM) conditioned generators by replacing text-only conditioning with final-layer hidden states extracted from visual pages, exploiting the fact that the frozen VLM already maps both text and images into the generator's conditioning space. On GenEval, V2V-Zero reaches 0.85 with a frozen Qwen-Image backbone, closely matching its optimized text-to-image performance without fine-tuning. To evaluate the broader V2V space, we introduce \textbf{Simple-V2V Bench}, spanning seven visual-conditioning tasks and seven models, including GPT Image 2, Nano Banana 2, Seedream 5.0 Lite, open-weight baselines, and a video extension. V2V-Zero scores 32.7/100, outperforming evaluated open-weight image baselines and revealing a clear capability hierarchy: attribute binding is strong, content generation is unreliable, and structural control remains hard even for commercial systems. A HunyuanVideo-1.5 extension scores 20.2/100, showing the interface transfers beyond images. Mechanistic analysis shows the default reasoning path is primarily visually routed, with 95.0\% of conditioning-token attention mass on visual-page hidden states.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑