Single image generation (SIG), described as generating diverse samples that have similar visual content with the given single image, is first introduced by SinGAN which builds a pyramid of GANs to progressively learn the internal patch distribution of the single image. It also shows great potentials in a wide range of image manipulation tasks. However, the paradigm of SinGAN has limitations in terms of generation quality and training time. Firstly, due to the lack of high-level information, SinGAN cannot handle the object images well as it does on the scene and texture images. Secondly, the separate progressive training scheme is time-consuming and easy to cause artifact accumulation. To tackle these problems, in this paper, we dig into the SIG problem and improve SinGAN by fully-utilization of internal and external priors. The main contributions of this paper include: 1) We introduce to SIG a regularized latent variable model. To the best of our knowledge, it is the first time to give a clear formulation and optimization goal of SIG, and all the existing methods for SIG can be regarded as special cases of this model. 2) We design a novel Prior-based end-to-end training GAN (PetsGAN) to overcome the problems of SinGAN. Our method gets rid of the time-consuming progressive training scheme and can be trained end-to-end. 3) We construct abundant qualitative and quantitative experiments to show the superiority of our method on both generated image quality, diversity, and the training speed. Moreover, we apply our method to other image manipulation tasks (e.g., style transfer, harmonization), and the results further prove the effectiveness and efficiency of our method.
Zhejiang University（浙江大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；National and Kapodistrian University of Athens（雅典国立卡波迪斯特里安大学）；Aalborg University（奥尔堡大学）；Cyprus University of Technology（塞浦路斯理工大学）
Sparse modeling for signal processing and machine learning has been at the focus of scientific research for over two decades. Among others, supervised sparsity-aware learning comprises two major paths paved by: a) discriminative methods and b) generative methods. The latter, more widely known as Bayesian methods, enable uncertainty evaluation w.r.t. the performed predictions. Furthermore, they can better exploit related prior information and naturally introduce robustness into the model, due to their unique capacity to marginalize out uncertainties related to the parameter estimates. Moreover,…
ReImagine: Rethinking Controllable High-Quality Human Video Generation via Image-First Synthesis
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）科学与工程学院）；Future Network of Intelligence Institute, CUHK-Shenzhen, China（CUHK-深圳智能网络研究院）
DiT-JSCC: Rethinking Deep JSCC with Diffusion Transformers and Semantic Representations
Beijing University of Posts and Telecommunications（北京邮电大学）；Shanghai Jiao Tong University（上海交通大学）；University of Shanghai for Science and Technology（上海科技大学）
National University of Singapore（新加坡国立大学）；Tencent PCG（腾讯平台与内容事业群）
Vector-Quantized (VQ-based) generative models usually consist of two basic components, i.e., VQ tokenizers and generative transformers. Prior research focuses on improving the reconstruction fidelity of VQ tokenizers but rarely examines how the improvement in reconstruction affects the generation ability of generative transformers. In this paper, we surprisingly find that improving the reconstruction fidelity of VQ tokenizers does not necessarily improve the generation. Instead, learning to compress semantic features within VQ tokenizers significantly improves generative transformers' ability …
Rethinking Diffusion Models with Symmetries through Canonicalization with Applications to Molecular Graph Generation
Massachusetts Institute of Technology（麻省理工学院）；Zhejiang University（浙江大学）；Peking University（北京大学）；Georgia Institute of Technology（佐治亚理工学院）；Princeton University（普林斯顿大学）；University of California, San Diego（加州大学圣地亚哥分校）
SPAC-Net: Rethinking Point Cloud Completion with Structural Prior
School of Digital Media Technology, Hangzhou Dianzi University（杭州电子科技大学数字媒体技术学院）；School of Informatics, Xiamen University（厦门大学信息学院）；School of Information, Renmin University of China（中国人民大学信息学院）
Vision-and-language navigation (VLN) is a trending topic which aims to navigate an intelligent agent to an expected position through natural language instructions. This work addresses the task of VLN from a previously-ignored aspect, namely the spatial route prior of the navigation scenes. A critically enabling innovation of this work is explicitly considering the spatial route prior under several different VLN settings. In a most information-rich case of knowing environment maps and admitting shortest-path prior, we observe that given an origin-destination node pair, the internal route can be…
LG AI Research（LG AI研究院）；Seoul National University（首尔大学）；GIST（韩国科学技术院）
Deep image prior (DIP) serves as a good inductive bias for diverse inverse problems. Among them, denoising is known to be particularly challenging for the DIP due to noise fitting with the requirement of an early stopping. To address the issue, we first analyze the DIP by the notion of effective degrees of freedom (DF) to monitor the optimization progress and propose a principled stopping criterion before fitting to noise without access of a paired ground truth image for Gaussian noise. We also propose the `stochastic temporal ensemble (STE)' method for incorporating techniques to further impr…
Cameras were originally designed using physics-based heuristics to capture aesthetic images. In recent years, there has been a transformation in camera design from being purely physics-driven to increasingly data-driven and task-specific. In this paper, we present a framework to understand the building blocks of this nascent field of end-to-end design of camera hardware and algorithms. As part of this framework, we show how methods that exploit both physics and data have become prevalent in imaging and computer vision, underscoring a key trend that will continue to dominate the future of task-…
National University of Singapore（新加坡国立大学）；University of Cambridge（剑桥大学）
Indirect image registration is a promising technique to improve image reconstruction quality by providing a shape prior for the reconstruction task. In this paper, we propose a novel hybrid method that seeks to reconstruct high quality images from few measurements whilst requiring low computational cost. With this purpose, our framework intertwines indirect registration and reconstruction tasks is a single functional. It is based on two major novelties. Firstly, we introduce a model based on deep nets to solve the indirect registration problem, in which the inversion and registration mappings …
LanP: Rethinking the Impact of Language Priors in Large Vision-Language Models
The Pennsylvania State University（宾夕法尼亚州立大学）；Singapore University of Technology and Design（新加坡科技设计大学）；Peking University（北京大学）；Rensselaer Polytechnic Institute（伦斯勒理工学院）
State-of-the-art atmospheric turbulence image restoration methods utilize standard image processing tools such as optical flow, lucky region and blind deconvolution to restore the images. While promising results have been reported over the past decade, many of the methods are agnostic to the physical model that generates the distortion. In this paper, we revisit the turbulence restoration problem by analyzing the reference frame generation and the blind deconvolution steps in a typical restoration pipeline. By leveraging tools in large deviation theory, we rigorously prove the minimum number o…
Recently, the concept of unsupervised learning for superpixel segmentation via CNNs has been studied. Essentially, such methods generate superpixels by convolutional neural network (CNN) employed on a single image, and such CNNs are trained without any labels or further information. Thus, such approach relies on the incorporation of priors, typically by designing an objective function that guides the solution towards a meaningful superpixel segmentation. In this paper we propose three key elements to improve the efficacy of such networks: (i) the similarity of the \emph{soft} superpixelated im…
Point cloud analysis is challenging due to irregularity and unordered data structure. To capture the 3D geometries, prior works mainly rely on exploring sophisticated local geometric extractors using convolution, graph, or attention mechanisms. These methods, however, incur unfavorable latency during inference, and the performance saturates over the past few years. In this paper, we present a novel perspective on this task. We notice that detailed local geometrical information probably is not the key to point cloud analysis -- we introduce a pure residual MLP network, called PointMLP, which in…
Rethinking the Adaptation of Vision Foundation Models for Efficient Cell Segmentation
School of Computer Science, University of Nottingham, UK（英国诺丁汉大学计算机科学学院）；School of Computer Science, University of Nottingham Ningbo China, China（中国宁波诺丁汉大学计算机科学学院）；School of Engineering and Physical Science, University of Lincoln, UK（英国林肯大学工程与物理科学学院）；Department of Computer Science, University of Rochester, USA（美国罗切斯特大学计算机科学系）
Rethinking Dense Optical Flow without Test-Time Scaling
Visual and Spatial AI Lab（视觉与空间人工智能实验室）；VCCM Section College of PVFA（VCCM学院光电工程学院）；Department of ECEN（电子工程系）；Department of CSCE（计算机科学与工程系）；Texas A&M University（德克萨斯A&M大学）
Amazon Web Services（亚马逊云科技）；University of Pennsylvania（宾夕法尼亚大学）；University of California, Los Angeles（加州大学洛杉矶分校）
Fine-tuning from pre-trained ImageNet models has become the de-facto standard for various computer vision tasks. Current practices for fine-tuning typically involve selecting an ad-hoc choice of hyperparameters and keeping them fixed to values normally used for training from scratch. This paper re-examines several common practices of setting hyperparameters for fine-tuning. Our findings are based on extensive empirical evaluation for fine-tuning on various transfer learning benchmarks. (1) While prior works have thoroughly investigated learning rate and batch size, momentum for fine-tuning is …
Continual Alignment for SAM: Rethinking Foundation Models for Medical Image Segmentation in Continual Learning
Xi’an Jiaotong University（西安交通大学）；School of Information and Communications Engineering, Xi’an Jiaotong University（信息与通信工程学院）；School of Mathematics and Statistics, Xi’an Jiaotong University（数学与统计学学院）
Rethinking Occlusion in FER: A Semantic-Aware Perspective and Go Beyond
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Department of Computer Science, Hong Kong Baptist University（香港 Baptist大学计算机科学系）
针对遮挡 FER 中特征提取失效与数据集偏差问题，本文提出 ORSANet，融合语义分割与关键点先验，并通过 MCM 和 DARELoss 提升相似表情判别力，同时构建 Occlu-FER 数据集，取得 SOTA 性能。
Rethinking Conditional Generation for Underwater Salient Object Detection
School of Computer Science and Technology, Hainan University（海南大学计算机科学与技术学院）；Key Laboratory of Machine Intelligence and System Control, Ministry of Education, Shandong University（山东大学机器智能与系统控制教育部重点实验室）；School of Data Science, Lingnan University（岭南大学数据科学学院）
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；Australian Institute for Machine Learning, The University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）
We study the composition style in deep image matting, a notion that characterizes a data generation flow on how to exploit limited foregrounds and random backgrounds to form a training dataset. Prior art executes this flow in a completely random manner by simply going through the foreground pool or by optionally combining two foregrounds before foreground-background composition. In this work, we first show that naive foreground combination can be problematic and therefore derive an alternative formulation to reasonably combine foregrounds. Our second contribution is an observation that matting…
Huazhong University of Science and Technology（华中科技大学）；Hong Kong University of Science and Technology（香港科技大学）
Vision transformers have recently set off a new wave in the field of medical image analysis due to their remarkable performance on various computer vision tasks. However, recent hybrid-/transformer-based approaches mainly focus on the benefits of transformers in capturing long-range dependency while ignoring the issues of their daunting computational complexity, high training costs, and redundant dependency. In this paper, we propose to employ adaptive pruning to transformers for medical image segmentation and propose a lightweight and effective hybrid network APFormer. To our best knowledge, …
Rethinking Theoretical Illumination for Efficient Low-Light Image Enhancement
Department of Electronic Engineering, Chung Yuan Christian University（Chung Yuan Christian University电子工程系）；Department of Electrical Engineering, University of Wisconsin–Milwaukee（University of Wisconsin–Milwaukee电气工程系）；Institute of Computer Science and Engineering, National Yang Ming Chiao Tung University（National Yang Ming Chiao Tung University计算机科学与工程研究所）
