StableSketcher: 通过视觉问答反馈增强扩散模型以生成基于像素的素描
StableSketcher: Enhancing Diffusion Model for Pixel-based Sketch Generation via Visual Question Answering Feedback
AI总结：
本文提出StableSketcher框架，通过优化潜在解码和视觉问答反馈提升扩散模型生成手绘素描的逼真度和语义一致性。
摘要：
尽管扩散模型在图像生成质量上取得了显著进步，但生成基于像素的人工绘制素描仍面临挑战。为此，我们提出StableSketcher，一种新的框架，使扩散模型能够生成高提示保真的手绘素描。在此框架中，我们微调变分自编码器以优化潜在解码，使其更好地捕捉素描的特征。同时，我们整合了基于视觉问答的新型奖励函数用于强化学习，从而提升文本-图像对齐和语义一致性。大量实验表明，StableSketcher生成的素描在风格保真度上有所提升，比Stable Diffusion基线更符合提示。此外，我们引入SketchDUO，据我们所知，首个包含实例级素描及其配对描述和问答对的数据集，从而解决现有数据集依赖图像-标签对的局限性。我们的代码和数据集将在接受后公开发布。项目页面：https://zihos.github.io/StableSketcher
英文摘要：
Although recent advancements in diffusion models have significantly enriched the quality of generated images, challenges remain in synthesizing pixel-based human-drawn sketches, a representative example of abstract expression. To combat these challenges, we propose StableSketcher, a novel framework that empowers diffusion models to generate hand-drawn sketches with high prompt fidelity. Within this framework, we fine-tune the variational autoencoder to optimize latent decoding, enabling it to better capture the characteristics of sketches. In parallel, we integrate a new reward function for reinforcement learning based on visual question answering, which improves text-image alignment and semantic consistency. Extensive experiments demonstrate that StableSketcher generates sketches with improved stylistic fidelity, achieving better alignment with prompts compared to the Stable Diffusion baseline. Additionally, we introduce SketchDUO, to the best of our knowledge, the first dataset comprising instance-level sketches paired with captions and question-answer pairs, thereby addressing the limitations of existing datasets that rely on image-label pairs. Our code and dataset will be made publicly available upon acceptance. Project page: https://zihos.github.io/StableSketcher
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
