DiffCap：探索连续扩散在图像描述生成中的应用
DiffCap: Exploring Continuous Diffusion on Image Captioning
AI总结：
本文提出DiffCap方法，首次将连续扩散模型应用于图像描述生成，通过自然转换离散词元并融合图像特征实现非自回归解码，在COCO数据集上以更简单结构取得与以往工作相当的结果，并显著提升了生成多样性。
摘要：
当前的图像描述生成工作通常侧重于以自回归方式生成描述。然而，专注于以非自回归方式生成描述的研究十分有限，而非自回归方式能带来更多的解码多样性。受扩散模型在生成逼真图像方面取得成功的启发，我们提出了一种将连续扩散应用于图像描述生成的新方法DiffCap。与输出为固定大小且连续的图像生成不同，图像描述的长度随离散词元而变化。我们的方法以自然的方式转换离散词元，并对其应用连续扩散，从而成功融合提取的图像特征以进行扩散描述生成。我们在COCO数据集上的实验表明，我们的方法使用更简单的结构即可取得与以往非自回归工作相当的结果。除质量外，DiffCap的一个有趣特性是其在生成过程中的高多样性，这是许多自回归模型所缺乏的。我们相信，我们在扩散语言生成中融合多模态特征的方法，因其简单性和解码灵活性，将激发更多关于多模态语言生成任务的研究。
英文摘要：
Current image captioning works usually focus on generating descriptions in an autoregressive manner. However, there are limited works that focus on generating descriptions non-autoregressively, which brings more decoding diversity. Inspired by the success of diffusion models on generating natural-looking images, we propose a novel method DiffCap to apply continuous diffusions on image captioning. Unlike image generation where the output is fixed-size and continuous, image description length varies with discrete tokens. Our method transforms discrete tokens in a natural way and applies continuous diffusion on them to successfully fuse extracted image features for diffusion caption generation. Our experiments on COCO dataset demonstrate that our method uses a much simpler structure to achieve comparable results to the previous non-autoregressive works. Apart from quality, an intriguing property of DiffCap is its high diversity during generation, which is missing from many autoregressive models. We believe our method on fusing multimodal features in diffusion language generation will inspire more researches on multimodal language generation tasks for its simplicity and decoding flexibility.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
