We introduce SemanticDraw, a new paradigm of interactive content creation where high-quality images are generated in near real-time from given multiple hand-drawn regions, each encoding prescribed semantic meaning. In order to maximize the productivity of content creators and to fully realize their artistic imagination, it requires both quick interactive interfaces and fine-grained regional controls in their tools. Despite astonishing generation quality from recent diffusion models, we find that existing approaches for regional controllability are very slow (52 seconds for $512 \times 512$ image) while not compatible with acceleration methods such as LCM, blocking their huge potential in interactive content creation. From this observation, we build our solution for interactive content creation in two steps: (1) we establish compatibility between region-based controls and acceleration techniques for diffusion models, maintaining high fidelity of multi-prompt image generation with $\times 10$ reduced number of inference steps, (2) we increase the generation throughput with our new multi-prompt stream batch pipeline, enabling low-latency generation from multiple, region-based text prompts on a single RTX 2080 Ti GPU. Our proposed framework is generalizable to any existing diffusion models and acceleration schedulers, allowing sub-second (0.64 seconds) image content creation application upon well-established image diffusion models. Our project page is: https://jaerinlee.com/research/semantic-draw
TaleForge: Interactive Multimodal System for Personalized Story Creation
1 University of Science, VNU-HCM, Ho Chi Minh City, Vietnam;2 Vietnam National University, Ho Chi Minh City, Vietnam;3 Department of Computer Science \ of Dayton Ohio, United States
InTeX: Interactive Text-to-texture Synthesis via Unified Depth-aware Inpainting
National Key Lab of General AI, Peking University(北京大学通用人工智能重点实验室);Zhejiang University(浙江大学);Skywork AI(天工AI);S-Lab, Nanyang Technological University(南洋理工大学S实验室)
iControl3D: An Interactive System for Controllable 3D Scene Generation
School of AIA, Huazhong University of Science and Technology(华中科技大学AIA学院);S-Lab, Nanyang Technological University(南洋理工大学S实验室);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);School of EIC, Huazhong University of Science and Technology(华中科技大学EIC学院);SenseTime Research(商汤科技研究院)
ContextMaster: Interactive Multi-Shot Video Creation via Fixed-Budget Sparse Context Routing
Tsinghua University(清华大学);Nanjing University(南京大学);Kling Team, Kuaishou Technology(快手科技影团队);ShanghaiTech University(上海科技大学);The Hong Kong University of Science and Technology(香港科技大学)
Image Conductor: Precision Control for Interactive Video Synthesis
Peking University(北京大学);ARC Lab, Tencent PCG(腾讯平台与内容事业群ARC实验室);Nanyang Technological University(南洋理工大学);Tsinghua University(清华大学);University of Macau(澳门大学);Shenzhen Institute of Advanced Technology(深圳先进技术研究院)
A New Algorithm for Interactive Structural Image Segmentation
IME-USP(圣保罗大学数学与统计研究所);University of São Paulo(圣保罗大学);Methodist University of Piracicaba(皮拉西卡巴卫理公会大学);UNIMEP(皮拉西卡巴卫理公会大学);Ecole Nationale Supérieure des Télécommunications(国立高等电信学院)