arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新，休息一下吧，周末愉快~~
arXiv 2403.04997cs.CLcs.CV
收藏

DiffChat：学习与文本到图像合成模型“聊天”以实现交互式图像创作

DiffChat: Learning to Chat with Text-to-Image Synthesis Models for Interactive Image Creation

South China University of Technology（华南理工大学）；Alibaba Group（阿里巴巴集团）

Jiapeng Wang, Chengyu Wang, Tingfeng Cao, Jun Huang, Lianwen Jin

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究提出DiffChat方法，通过构建InstructPE数据集监督训练、结合三标准反馈的强化学习框架，实现LLM与文本到图像合成模型的交互式创作对话，生成图像质量优于基线与竞品。

摘要：

我们提出了DiffChat，这是一种将大语言模型（LLM）对齐以与以提示词为输入的文本到图像合成（TIS）模型（例如Stable Diffusion）“聊天”、用于交互式图像创作的新方法。给定原始提示词/图像和用户指定的指令，DiffChat能有效做出适当修改并生成目标提示词，可用于创建高质量的目标图像。为实现这一点，我们首先收集了一个名为InstructPE的指令跟随提示工程数据集，用于DiffChat的监督训练。接着，我们提出一个强化学习框架，以图像创作的三个核心标准即美学、用户偏好、内容完整性作为反馈。该框架包含一种动作空间动态修改技术，可在离策略采样期间获取更相关的正样本和更难的负样本。我们还将内容完整性引入价值估计函数，以进一步提升生成图像的质量。基于自动评估和人工评估，我们的方法表现出优于基线模型和强劲竞品的性能，充分证明了其有效性。

英文摘要：

We present DiffChat, a novel method to align Large Language Models (LLMs) to "chat" with prompt-as-input Text-to-Image Synthesis (TIS) models (e.g., Stable Diffusion) for interactive image creation. Given a raw prompt/image and a user-specified instruction, DiffChat can effectively make appropriate modifications and generate the target prompt, which can be leveraged to create the target image of high quality. To achieve this, we first collect an instruction-following prompt engineering dataset named InstructPE for the supervised training of DiffChat. Next, we propose a reinforcement learning framework with the feedback of three core criteria for image creation, i.e., aesthetics, user preference, and content integrity. It involves an action-space dynamic modification technique to obtain more relevant positive samples and harder negative samples during the off-policy sampling. Content integrity is also introduced into the value estimation function for further improvement of produced images. Our method can exhibit superior performance than baseline models and strong competitors based on both automatic and human evaluations, which fully demonstrates its effectiveness.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑