VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control
University of Toronto（多伦多大学）；Vector Institute（矢量研究所）；Snap Inc.（斯纳普公司）；SFU（西蒙菲莎大学）
Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin-Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagl… 展开作者
Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin-Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, David B. Lindell, Sergey Tulyakov
Modern text-to-video synthesis models demonstrate coherent, photorealistic generation of complex videos from a text description. However, most existing models lack fine-grained control over camera movement, which is critical for downstream applications related to content creation, visual effects, and 3D vision. Recently, new methods demonstrate the ability to generate videos with controllable camera poses these techniques leverage pre-trained U-Net-based diffusion models that explicitly disentangle spatial and temporal generation. Still, no existing approach enables camera control for new, transformer-based video diffusion models that process spatial and temporal information jointly. Here, we propose to tame video transformers for 3D camera control using a ControlNet-like conditioning mechanism that incorporates spatiotemporal camera embeddings based on Plücker coordinates. The approach demonstrates state-of-the-art performance for controllable video generation after fine-tuning on the RealEstate10K dataset. To the best of our knowledge, our work is the first to enable camera control for transformer-based video diffusion models.
Heidelberg Collaboratory for Image Processing（海德堡图像处理合作实验室）；Heidelberg University（海德堡大学）；IWR（跨学科科学计算中心）
Designed to learn long-range interactions on sequential data, transformers continue to show state-of-the-art results on a wide variety of tasks. In contrast to CNNs, they contain no inductive bias that prioritizes local interactions. This makes them expressive, but also computationally infeasible for long sequences, such as high-resolution images. We demonstrate how combining the effectiveness of the inductive bias of CNNs with the expressivity of transformers enables them to model and thereby synthesize high-resolution images. We show how to (i) use CNNs to learn a context-rich vocabulary of …
UniGP: Taming Diffusion Transformer for Prior-Preserved Unified Generation and Perception
The Hong Kong University of Science and Technology（香港科学与技术大学）；DAMO Academy, Alibaba Group（阿里云达摩院）；Hupan Lab（虎扑实验室）；Zhejiang University（浙江大学）；Tsinghua University（清华大学）；The Chinese University of Hong Kong（香港中文大学）；Zeekr Automobile R&D Co., Ltd.（小鹏汽车研发有限公司）
UniMC: Taming Diffusion Transformer for Unified Keypoint-Guided Multi-Class Image Generation
The Hong Kong University of Science and Technology（香港科学与技术大学）；The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）；Peking University（北京大学）；Tencent（腾讯）；National University of Singapore（新加坡国立大学）
MV-Performer: Taming Video Diffusion Model for Faithful and Synchronized Multi-view Performer Synthesis
Great Bay University（大亚湾大学）；School of Artificial Intelligence, Shenzhen University（深圳大学人工智能学院）；Guangdong Provincial Key Laboratory of Future Networks of Intelligence（广东省未来网络智能化重点实验室）
ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis
Peking University（北京大学）；Peng Cheng Laboratory（鹏城实验室）；The Chinese University of Hong Kong（香港中文大学）；Tencent AI Lab（腾讯AI Lab）；ARC Lab, Tencent PCG（腾讯PCG ARC实验室）；Hong Kong University of Science and Technology（香港科技大学）；Monash University（莫纳什大学）
VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Huazhong University of Science and Technology（华中科技大学）；Shenzhen Loop Area Institute（深圳河套学院）；University of Science and Technology of China（中国科学技术大学）
DOME: Taming Diffusion Model into High-Fidelity Controllable Occupancy World Model
Horizon Robotics；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；Institute of Automation Chinese Academy of Sciences（中国科学院自动化研究所）；The University of Hong Kong（香港大学）；University of Pennsylvania（宾夕法尼亚大学）；HKUST(GZ)（香港科技大学（广州））；University of Chinese Academy of Sciences（中国科学院大学）；HKUST（香港科技大学）
The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）
Animating virtual avatars to make co-speech gestures facilitates various applications in human-machine interaction. The existing methods mainly rely on generative adversarial networks (GANs), which typically suffer from notorious mode collapse and unstable training, thus making it difficult to learn accurate audio-gesture joint distributions. In this work, we propose a novel diffusion-based framework, named Diffusion Co-Speech Gesture (DiffGesture), to effectively capture the cross-modal audio-to-gesture associations and preserve temporal coherence for high-fidelity audio-driven co-speech gest…
DP-aware AdaLN-Zero: Taming Conditioning-Induced Heavy-Tailed Gradients in Differentially Private Diffusion
School of Computer Science（计算机科学学院）；Big Data, Minjiang University, Fuzhou, Fujian, China.（大数据，闽江大学，福州，福建，中国）；School of Information, Renmin University of China, Beijing, China（信息学院，中国人民大学，北京，中国）
Recent advances in visually-induced audio generation are based on sampling short, low-fidelity, and one-class sounds. Moreover, sampling 1 second of audio from the state-of-the-art model takes minutes on a high-end GPU. In this work, we propose a single model capable of generating visually relevant, high-fidelity sounds prompted with a set of frames from open-domain videos in less time than it takes to play it on a single GPU. We train a transformer to sample a new spectrogram from the pre-trained spectrogram codebook given the set of video features. The codebook is obtained using a variant of…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Google Research（谷歌研究院）
Convolutional Neural Networks (CNN) have dominated the field of detection ever since the success of AlexNet in ImageNet classification [12]. With the sweeping reform of Transformers [27] in natural language processing, Carion et al. [2] introduce the Transformer-based detection method, i.e., DETR. However, due to the quadratic complexity in the self-attention mechanism in the Transformer, DETR is never able to incorporate multi-scale features as performed in existing CNN-based detectors, leading to inferior results in small object detection. To mitigate this issue and further improve performan…
Camera pose estimation or camera relocalization is the centerpiece in numerous computer vision tasks such as visual odometry, structure from motion (SfM) and SLAM. In this paper we propose a neural network approach with a graph transformer backbone, namely TransCamP, to address the camera relocalization problem. In contrast with prior work where the pose regression is mainly guided by photometric consistency, TransCamP effectively fuses the image features, camera pose information and inter-frame relative camera motions into encoded graph attributes and is trained towards the graph consistency …
Single image camera calibration is the task of estimating the camera parameters from a single input image, such as the vanishing points, focal length, and horizon line. In this work, we propose Camera calibration TRansformer with Line-Classification (CTRL-C), an end-to-end neural network-based approach to single image camera calibration, which directly estimates the camera parameters from an image and a set of line segments. Our network adopts the transformer architecture to capture the global structure of an image with multi-modal inputs in an end-to-end manner. We also propose an auxiliary t…
The task of video virtual try-on aims to fit the target clothes to a person in the video with spatio-temporal consistency. Despite tremendous progress of image virtual try-on, they lead to inconsistency between frames when applied to videos. Limited work also explored the task of video-based virtual try-on but failed to produce visually pleasing and temporally coherent results. Moreover, there are two other key challenges: 1) how to generate accurate warping when occlusions appear in the clothing region; 2) how to generate clothes and non-target body parts (e.g. arms, neck) in harmony with the…
ThermalLoc: A Vision Transformer-Based Approach for Robust Thermal Camera Relocalization in Large-Scale Environments
College of Intelligence Science and Technology, National University of Defense Technology（智能科学与技术学院，国防科技大学）；China Academy of Launch Vehicle Technology（航天科技研究院）；PEAK-Lab, The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）PEAK实验室）
