The advent of text-driven 360-degree panorama generation, enabling the synthesis of 360-degree panoramic images directly from textual descriptions, marks a transformative advancement in immersive visual content creation. This innovation significantly simplifies the traditionally complex process of producing such content. Recent progress in text-to-image diffusion models has accelerated the rapid development in this emerging field. This survey presents a comprehensive review of text-driven 360-degree panorama generation, offering an in-depth analysis of state-of-the-art algorithms. We extend our analysis to two closely related domains: text-driven 360-degree 3D scene generation and text-driven 360-degree panoramic video generation. Furthermore, we critically examine current limitations and propose promising directions for future research. A curated project page with relevant resources and research papers is available at https://littlewhitesea.github.io/Text-Driven-Pano-Gen/.
What Makes for Text to 360-degree Panorama Generation with Stable Diffusion?
Australian National University（澳大利亚国立大学）；The University of Hong Kong（香港大学）；Beijing Innovation Center of Humanoid Robotics（北京人形机器人创新中心）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Programmable-Room: Interactive Textured 3D Room Meshes Generation Empowered by Large Language Models
Department of Electronic Engineering, Sogang University（电子工程系，ソガン大学）；Department of Electrical and Electronics Engineering, Pusan National University（电气电子工程系，釜山国立大学）
We present an end-to-end deep learning framework for indoor panoramic image inpainting. Although previous inpainting methods have shown impressive performance on natural perspective images, most fail to handle panoramic images, particularly indoor scenes, which usually contain complex structure and texture content. To achieve better inpainting quality, we propose to exploit both the global and local context of indoor panorama during the inpainting process. Specifically, we take the low-level layout edges estimated from the input panorama as a prior to guide the inpainting model for recovering …
Conditional Panoramic Image Generation via Masked Autoregressive Modeling
School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；Insta360 Research（Insta360研究院）；National University of Singapore（新加坡国立大学）；The University of Tokyo（东京大学）
TiP4GEN: Text to Immersive Panorama 4D Scene Generation
Institute of Information Science, Beijing Jiaotong University（信息科学研究院，北京交通大学）；Visual Intelligence + X International Joint Laboratory（视觉智能+X国际联合实验室）；University of Toronto（多伦多大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Image extension models have broad applications in image editing, computational photography and computer graphics. While image inpainting has been extensively studied in the literature, it is challenging to directly apply the state-of-the-art inpainting methods to image extension as they tend to generate blurry or repetitive pixels with inconsistent semantics. We introduce semantic conditioning to the discriminator of a generative adversarial network (GAN), and achieve strong results on image extension with coherent semantics and visually pleasing colors and textures. We also show promising res…
Recent advances in text-to-image generation with diffusion models present transformative capabilities in image quality. However, user controllability of the generated image, and fast adaptation to new tasks still remains an open challenge, currently mostly addressed by costly and long re-training and fine-tuning or ad-hoc adaptations to specific image generation tasks. In this work, we present MultiDiffusion, a unified framework that enables versatile and controllable image generation, using a pre-trained text-to-image diffusion model, without any further training or finetuning. At the center …
Matrix-3D: Omnidirectional Explorable 3D World Generation
Skywork AI；Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）
Georgia Institute of Technology（佐治亚理工学院）；NVIDIA Corporation（英伟达公司）
We present DiffCollage, a compositional diffusion model that can generate large content by leveraging diffusion models trained on generating pieces of the large content. Our approach is based on a factor graph representation where each factor node represents a portion of the content and a variable node represents their overlap. This representation allows us to aggregate intermediate outputs from diffusion models defined on individual nodes to generate content of arbitrary size and shape in parallel without resorting to an autoregressive generation procedure. We apply DiffCollage to various tas…
LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Zhejiang University（浙江大学）；Stanford University（斯坦福大学）；Nanyang Technological University（南洋理工大学）
PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection
Inria, École normale supérieure, CNRS, PSL Research University（法国国家信息与自动化研究所，巴黎高等师范学院，法国国家科学研究中心，巴黎文理研究大学）；Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague（捷克信息学、机器人与控制论研究所，布拉格捷克理工大学）
The richness of natural images makes the quest for optimal representations in image processing and computer vision challenging. The latter observation has not prevented the design of image representations, which trade off between efficiency and complexity, while achieving accurate rendering of smooth regions as well as reproducing faithful contours and textures. The most recent ones, proposed in the past decade, share an hybrid heritage highlighting the multiscale and oriented nature of edges and patterns in images. This paper presents a panorama of the aforementioned literature on decompositi…
PanoGen++: Domain-Adapted Text-Guided Panoramic Environment Generation for Vision-and-Language Navigation
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；Tianjin Artificial Intelligence Innovation Center（天津人工智能创新中心）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））
TU Delft（代尔夫特理工大学）；TU Eindhoven（埃因霍温理工大学）；Cyclomedia B.V（Cyclomedia公司）
The current paradigm in privacy protection in street-view images is to detect and blur sensitive information. In this paper, we propose a framework that is an alternative to blurring, which automatically removes and inpaints moving objects (e.g. pedestrians, vehicles) in street-view imagery. We propose a novel moving object segmentation algorithm exploiting consistencies in depth across multiple street-view images that are later combined with the results of a segmentation network. The detected moving objects are removed and inpainted with information from other views, to obtain a realistic out…
