We present a general-purpose framework for image modelling and vision tasks based on probabilistic frame prediction. Our approach unifies a broad range of tasks, from image segmentation, to novel view synthesis and video interpolation. We pair this framework with an architecture we term Transframer, which uses U-Net and Transformer components to condition on annotated context frames, and outputs sequences of sparse, compressed image features. Transframer is the state-of-the-art on a variety of video generation benchmarks, is competitive with the strongest models on few-shot view synthesis, and can generate coherent 30 second videos from a single image without any explicit geometric information. A single generalist Transframer simultaneously produces promising results on 8 tasks, including semantic segmentation, image classification and optical flow prediction with no task-specific architectural components, demonstrating that multi-task computer vision can be tackled using probabilistic image models. Our approach can in principle be applied to a wide range of applications that require learning the conditional structure of annotated image-formatted data.
A Sensor Agnostic Domain Generalization Framework for Leveraging Geospatial Foundation Models: Enhancing Semantic Segmentation viaSynergistic Pseudo-Labeling and Generative Learning
University of Houston(休斯敦大学);Purdue University(普渡大学)
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);Platform and Content Group (PCG), Tencent(腾讯平台与内容事业群);Shanghai AI Lab(上海人工智能实验室)
GLOWin: A Flow-based Invertible Generative Framework for Learning Disentangled Feature Representations in Medical Images
Technical University of Munich(慕尼黑工业大学);deepc GmbH(deepc有限公司);Ludwig Maximilians University Munich(慕尼黑大学);Kyung Hee University(庆熙大学);Johns Hopkins University(约翰斯·霍普金斯大学)
Thinking with Frames: Generative Video Distortion Evaluation via Frame Reward Model
University of Science and Technology of China(中国科学技术大学);Kling Team, Kuaishou Technology(快手科技 Kling 团队);Institute of Software Chinese Academy of Sciences(中国科学院软件研究所)
A Unified Generative Framework for Realistic Lidar Simulation in Autonomous Driving Systems
WMG, University of Warwick(华威大学WMG学院);School of Electronics, Electrical Engineering and Computer Science at Queen’s University of Belfast(贝尔法斯特女王大学电子、电气工程与计算机科学学院)