面向开放词汇360度图像生成的自回归全感知外扩方法
Autoregressive Omni-Aware Outpainting for Open-Vocabulary 360-Degree Image Generation
AI总结：
针对窄视场图像与文本条件生成360度图像时细节和提示一致性不足的问题，提出AOG-Net自回归全局-局部全感知外扩框架，实现开放词汇可控生成并达到SOTA。
摘要：
360度（全向）图像可提供场景的完整球形视图。近年来，从数码相机和智能手机拍摄的传统窄视场（NFoV）图像合成360度图像受到越来越多关注，以在虚拟现实等多种场景中提供沉浸式体验。然而，现有方法通常难以合成复杂视觉细节，或难以确保生成图像与用户提供的提示保持一致。本研究提出自回归全感知生成网络（AOG-Net），通过在NFoV和文本指导共同或单独作用下逐步外扩不完整的360度图像来实现360度图像生成。该自回归方案不仅能够通过动态生成和调整过程获得更细粒度且与文本一致的模式，还为用户提供在整个生成过程中编辑其条件的更大灵活性。本文设计了全局-局部条件机制，以全面构建每个自回归步骤中的外扩指导。文本指导、全向视觉线索、NFoV输入和全向几何信息被编码，并进一步通过基于交叉注意力的transformer组织为全局流和局部流，输入到条件化生成主干模型。由于AOG-Net能够兼容利用大规模模型作为条件编码器和生成先验，其生成过程可使用广泛的开放词汇文本指导。在两个常用的、涵盖室内和室外设置的360度图像数据集上进行的综合实验表明，所提方法达到了最先进性能。代码将公开发布。
英文摘要：
A 360-degree (omni-directional) image provides an all-encompassing spherical view of a scene. Recently, there has been an increasing interest in synthesising 360-degree images from conventional narrow field of view (NFoV) images captured by digital cameras and smartphones, for providing immersive experiences in various scenarios such as virtual reality. Yet, existing methods typically fall short in synthesizing intricate visual details or ensure the generated images align consistently with user-provided prompts. In this study, autoregressive omni-aware generative network (AOG-Net) is proposed for 360-degree image generation by out-painting an incomplete 360-degree image progressively with NFoV and text guidances joinly or individually. This autoregressive scheme not only allows for deriving finer-grained and text-consistent patterns by dynamically generating and adjusting the process but also offers users greater flexibility to edit their conditions throughout the generation process. A global-local conditioning mechanism is devised to comprehensively formulate the outpainting guidance in each autoregressive step. Text guidances, omni-visual cues, NFoV inputs and omni-geometry are encoded and further formulated with cross-attention based transformers into a global stream and a local stream into a conditioned generative backbone model. As AOG-Net is compatible to leverage large-scale models for the conditional encoder and the generative prior, it enables the generation to use extensive open-vocabulary text guidances. Comprehensive experiments on two commonly used 360-degree image datasets for both indoor and outdoor settings demonstrate the state-of-the-art performance of our proposed method. Our code will be made publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
