发表机构
University of Technology Sydney; Geely; City University of Hong Kong; City University of Macau（悉尼科技大学; 吉利; 香港城市大学; 澳门城市大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究针对现有文本到图像扩散模型提示逆推方法的局限，提出联合恢复语义提示与潜在噪声的Dualin方法，实现了高质量逆推提示与最优图像保真度，为可控图像编辑奠定基础。
AI 中文摘要
提示逆推作为一种典型的逆向工程技术，可让文本到图像（T2I）扩散模型无需大量提示工程即可生成所需目标图像。然而，现有提示逆推方法存在显著局限：（1）基于梯度的方法不稳定且不可解释，常导致生成图像出现严重伪影；（2）无梯度方法虽能生成人类可读的提示，但因缺乏细粒度细节对齐，仍无法保持视觉保真度。我们认为这些局限源于将提示逆推视为逆向工程的充分条件，忽视了编码结构信息的潜在噪声的关键作用。因此，我们提出Dualin（双重逆推），一种联合恢复目标图像语义提示与潜在噪声的两阶段方法。第一阶段，我们整合视觉语言模型CLIP与大语言模型，逆推出忠实且人类可解释的硬提示；第二阶段，无条件DDIM逆推重构目标图像的精确潜在噪声，保证结构信息层面的一致性。理论上，我们证明逆推所得噪声无需重新优化即可实现灵活的图像编辑。在多样化数据集上的大量实验表明，Dualin同时生成高质量逆推提示并达到当前最优的图像保真度，此外还能为精确且可控的图像编辑建立可靠基础。
英文摘要
Prompt inversion, as a typical reverse engineering technique, enables text-to-image (T2I) diffusion models to generate the desired target images without extensive prompt engineering. However, existing prompt inversion methods suffer from significant limitations: (1) gradient-based methods are unstable and uninterpretable, often resulting in generated images with severe artifacts; (2) gradient-free methods yield human-readable prompts but still fail to preserve visual fidelity due to the lack of fine-grained detail alignment. We contend that the limitations stem from treating prompt inversion as a sufficient condition for reverse engineering, ignoring the critical role of the latent noise that encodes structural information. Consequently, we propose Dualin (Dual inversion), a two-stage method that jointly recovers both the semantic prompt and latent noise of the target image. In the first stage, we integrate vision-language model, CLIP and large language model to invert a faithful, human-interpretable hard prompt. In the second stage, unconditional DDIM inversion reconstructs the exact latent noise of the target image, guaranteeing the consistency at the structural information level. Theoretically, we prove that the inverted noise enables flexible image editing without re-optimization. Extensive experiments on diverse datasets demonstrate that Dualin simultaneously generates high-quality inverted prompts and achieves state-of-the-art image fidelity. Additionally, Dualin can establish a robust foundation for the precise and controllable image editing.
CommentsAccepted by ACM MM 2026
