Video prediction is a pixel-wise dense prediction task to infer future frames based on past frames. Missing appearance details and motion blur are still two major problems for current predictive models, which lead to image distortion and temporal inconsistency. In this paper, we point out the necessity of exploring multi-frequency analysis to deal with the two problems. Inspired by the frequency band decomposition characteristic of Human Vision System (HVS), we propose a video prediction network based on multi-level wavelet analysis to deal with spatial and temporal information in a unified manner. Specifically, the multi-level spatial discrete wavelet transform decomposes each video frame into anisotropic sub-bands with multiple frequencies, helping to enrich structural information and reserve fine details. On the other hand, multi-level temporal discrete wavelet transform which operates on time axis decomposes the frame sequence into sub-band groups of different frequencies to accurately capture multi-frequency motions under a fixed frame rate. Extensive experiments on diverse datasets demonstrate that our model shows significant improvements on fidelity and temporal consistency over state-of-the-art works.
SegmentDreamer: Towards High-fidelity Text-to-3D Synthesis with Segmented Consistency Trajectory Distillation
Sun Yat-sen University（中山大学）；Great Bay University（大贝大学）；Pazhou Lab (Huangpu)（琶洲实验室）；Guangdong Province Key Laboratory of Information Security Technology（广东省信息安全技术重点实验室）
The Hong Kong University of Science and Technology（香港科技大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
This paper investigates the geometric consistency for monocular 3D object detection, which suffers from the ill-posed depth estimation. We first conduct a thorough analysis to reveal how existing methods fail to consistently localize objects when different geometric shifts occur. In particular, we design a series of geometric manipulations to diagnose existing detectors and then illustrate their vulnerability to consistently associate the depth with object apparent sizes and positions. To alleviate this issue, we propose four geometry-aware data augmentation approaches to enhance the geometric…
Exploring Semantic Consistency in Unpaired Image Translation to Generate Data for Surgical Applications
NCT/UCC Dresden（德累斯顿国家肿瘤疾病中心/德累斯顿肿瘤中心）；SECAI, TU Dresden（德累斯顿工业大学SECAI中心）；Faculty of medicine & University Hospital TU Dresden（德累斯顿工业大学医学院及大学医院）；CeTI, TU Dresden（德累斯顿工业大学CeTI中心）
University of Oxford（牛津大学）；University of Texas（德克萨斯大学）；Tsinghua University（清华大学）；Microsoft Research Asia（微软亚洲研究院）
Previous cycle-consistency correspondence learning methods usually leverage image patches for training. In this paper, we present a fully convolutional method, which is simpler and more coherent to the inference process. While directly applying fully convolutional training results in model collapse, we study the underline reason behind this collapse phenomenon, indicating that the absolute positions of pixels provide a shortcut to easily accomplish cycle-consistence, which hinders the learning of meaningful visual representations. To break this absolute position shortcut, we propose to apply d…
Boston University（波士顿大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
While transformers have greatly boosted performance in semantic segmentation, domain adaptive transformers are not yet well explored. We identify that the domain gap can cause discrepancies in self-attention. Due to this gap, the transformer attends to spurious regions or pixels, which deteriorates accuracy on the target domain. We propose to perform adaptation on attention maps with cross-domain attention layers that share features between the source and the target domains. Specifically, we impose consistency between predictions from cross-domain attention and self-attention modules to encour…
High-fidelity 3D Gaussian Inpainting: preserving multi-view consistency and photorealistic details
School of Information Science and Technology, Dalian Maritime University（信息科学与技术学院，大连海洋大学）；School of Science, Zhejiang Sci-Tech University（科学学院，浙江科技大学）
本文提出一种 3D Gaussian 修复框架，通过自动掩码精修和不确定性引导的区域级优化，利用稀疏修复视角重建完整 3D 场景，并在视觉质量与多视角一致性上超过现有方法。
MAC-Splat: Multi-Attribute Consistency for High-Fidelity Sparse-View Reconstruction
Xidian University（西安电子科技大学）；Harvard University（哈佛大学）；Nanyang Technological University（南洋理工大学）；City University of Hong Kong（香港城市大学）；Xi’an Jiaotong University（西安交通大学）
DICS: Exploring Data Intrinsic Consistency for Visual Instruction Selection
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Alibaba Token Hub, Alibaba Group（阿里巴巴集团）
University of Science and Technology of China（中国科学技术大学）；JD AI Research（京东人工智能研究院）；City University of Hong Kong（香港城市大学）；Peking University（北京大学）
Rendering synthetic data (e.g., 3D CAD-rendered images) to generate annotations for learning deep models in vision tasks has attracted increasing attention in recent years. However, simply applying the models learnt on synthetic images may lead to high generalization error on real images due to domain shift. To address this issue, recent progress in cross-domain recognition has featured the Mean Teacher, which directly simulates unsupervised domain adaptation as semi-supervised learning. The domain gap is thus naturally bridged with consistency regularization in a teacher-student scheme. In th…
FlowDreamer: Exploring High Fidelity Text-to-3D Generation via Rectified Flow
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Alibaba Group（阿里巴巴集团）；The Hong Kong University of Science and Technology（香港科技大学）
University of Science and Technology of China（中国科学技术大学）；Microsoft Cloud AI（微软云人工智能部门）；City University of Hong Kong（香港城市大学）；Wormpex AI Research（Wormpex人工智能研究院）
The intellectual property (IP) of Deep neural networks (DNNs) can be easily ``stolen'' by surrogate model attack. There has been significant progress in solutions to protect the IP of DNN models in classification tasks. However, little attention has been devoted to the protection of DNNs in image processing tasks. By utilizing consistent invisible spatial watermarks, one recent work first considered model watermarking for deep image processing networks and demonstrated its efficacy in many downstream tasks. Nevertheless, it highly depends on the hypothesis that the embedded watermarks in the n…
A key challenge for novel view synthesis of monocular portrait images is 3D consistency under continuous pose variations. Most existing methods rely on 2D generative models which often leads to obvious 3D inconsistency artifacts. We present a 3D-consistent novel view synthesis approach for monocular portrait images based on a recent proposed 3D-aware GAN, namely Generative Radiance Manifolds (GRAM), which has shown strong 3D consistency at multiview image generation of virtual subjects via the radiance manifolds representation. However, simply learning an encoder to map a real image into the l…
Null-Space Diffusion Distillation Unlocks Speed, Fidelity and Realism in Lensless Imaging
D3 Center, The University of Osaka（大阪大学D3中心）；The University of Osaka（大阪大学）；SANKEN, The University of Osaka（SANKEN与大阪大学）；Grad. Sch. of Eng. Sci., The University of Osaka（大阪大学工科科学院）
From Static to Dynamic: Exploring Self-supervised Image-to-Video Representation Transfer Learning
School of Computer Science and Technology, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所人工智能安全国家重点实验室）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）
Sora Generates Videos with Stunning Geometrical Consistency
Nankai University（南开大学）；ByteDance Inc（字节跳动公司）；Wuhan University（武汉大学）；Nankai International Advanced Research Institute (Shenzhen Futian)（南开大学国际高等研究院（深圳福田））
Cross-domain image-to-image translation should satisfy two requirements: (1) preserve the information that is common to both domains, and (2) generate convincing images covering variations that appear in the target domain. This is challenging, especially when there are no example translations available as supervision. Adversarial cycle consistency was recently proposed as a solution, with beautiful and creative results, yielding much follow-up work. However, augmented reality applications cannot readily use such techniques to provide users with compelling translations of real scenes, because t…
PositionIC: Unified Position and Identity Consistency for Image Customization
MeiGen AI Team, Meituan（美团MeiGen AI团队）；Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University（上海智能信息处理关键实验室，计算机科学与人工智能学院，复旦大学）；College of Intelligent Robotics and Advanced Manufacturing, Fudan University（智能机器人与先进制造学院，复旦大学）
SNUAILAB（首尔国立大学人工智能实验室）；Seoul National University（首尔国立大学）
Active Learning (AL) and Semi-supervised Learning are two techniques that have been studied to reduce the high cost of deep learning by using a small amount of labeled data and a large amount of unlabeled data. To improve the accuracy of models at a lower cost, we propose a method called Active Semi-supervised Learning (ASSL), which combines AL and SSL. To maximize the synergy between AL and SSL, we focused on the differences between ASSL and AL. ASSL involves more dynamic model updates than AL due to the use of unlabeled data in the training process, resulting in the temporal instability of t…
We present a recurrent network for the 3D reconstruction of neurons that sequentially generates binary masks for every object in an image with spatio-temporal consistency. Our network models consistency in two parts: (i) local, which allows exploring non-occluding and temporally-adjacent object relationships with bi-directional recurrence. (ii) non-local, which allows exploring long-range object relationships in the temporal domain with skip connections. Our proposed network is end-to-end trainable from an input image to a sequence of object masks, and, compared to methods relying on object bo…
Tsinghua University（清华大学）；Microsoft Research Asia（微软亚洲研究院）
Contrastive learning methods for unsupervised visual representation learning have reached remarkable levels of transfer performance. We argue that the power of contrastive learning has yet to be fully unleashed, as current methods are trained only on instance-level pretext tasks, leading to representations that may be sub-optimal for downstream tasks requiring dense pixel predictions. In this paper, we introduce pixel-level pretext tasks for learning dense feature representations. The first task directly applies contrastive learning at the pixel level. We additionally propose a pixel-to-propag…
Microsoft Research Asia（微软亚洲研究院）；University of Science and Technology of China（中国科学技术大学）
Cycle consistency is widely used for face editing. However, we observe that the generator tends to find a tricky way to hide information from the original image to satisfy the constraint of cycle consistency, making it impossible to maintain the rich details (e.g., wrinkles and moles) of non-editing areas. In this work, we propose a simple yet effective method named HifaFace to address the above-mentioned problem from two perspectives. First, we relieve the pressure of the generator to synthesize rich details by directly feeding the high-frequency information of the input image into the end of…
NuiWorld: Exploring a Scalable Framework for End-to-End Controllable World Generation
Simon Fraser University（西蒙弗雷泽大学）；National Yang Ming Chiao Tung University（国家阳明交通大学）；Simon Fraser University, CIFAR AI Chair, Amii（西蒙弗雷泽大学、CIFAR人工智能主席、Amii）
Enhancing Texture Generation with High-Fidelity Using Advanced Texture Priors
MetaApp AI Research（MetaApp AI研究院）；Zhengzhou University（郑州大学）；Beijing Institute of Technology（北京理工大学）；Beijing University of Technology（北京工业大学）；Northeastern University（东北大学）
