Existing color-guided depth super-resolution (DSR) approaches require paired RGB-D data as training samples where the RGB image is used as structural guidance to recover the degraded depth map due to their geometrical similarity. However, the paired data may be limited or expensive to be collected in actual testing environment. Therefore, we explore for the first time to learn the cross-modality knowledge at training stage, where both RGB and depth modalities are available, but test on the target dataset, where only single depth modality exists. Our key idea is to distill the knowledge of scene structural guidance from RGB modality to the single DSR task without changing its network architecture. Specifically, we construct an auxiliary depth estimation (DE) task that takes an RGB image as input to estimate a depth map, and train both DSR task and DE task collaboratively to boost the performance of DSR. Upon this, a cross-task interaction module is proposed to realize bilateral cross task knowledge transfer. First, we design a cross-task distillation scheme that encourages DSR and DE networks to learn from each other in a teacher-student role-exchanging fashion. Then, we advance a structure prediction (SP) task that provides extra structure regularization to help both DSR and DE networks learn more informative structure representations for depth recovery. Extensive experiments demonstrate that our scheme achieves superior performance in comparison with other DSR methods.
Efficient scene text image super-resolution with semantic guidance
School of Computer Science, Fudan University（复旦大学计算机科学学院）；Videt Technology（Videt科技）；Innovation Center of Calligraphy and Painting Creation Technology, MCT（MCT书画创作技术创新中心）
Restore Text First, Enhance Image Later: Two-Stage Scene Text Image Super-Resolution with Glyph Structure Guidance
VCIP, CS, Nankai University（南开大学计算机科学与技术学院）；vivo Mobile Communication Co. Ltd（vivo移动通信有限公司）；SDS, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）信息科学与技术学院）
Scene text image super-resolution (STISR) aims to improve the resolution and visual quality of low-resolution (LR) scene text images, and consequently boost the performance of text recognition. However, most of existing STISR methods regard text images as natural scene images, ignoring the categorical information of text. In this paper, we make an inspiring attempt to embed categorical text prior into STISR model training. Specifically, we adopt the character probability sequence as the text prior, which can be obtained conveniently from a text recognition model. The text prior provides catego…
Wuhan University（武汉大学）；National Institute of Informatics（国立信息学研究所）；National Tsing Hua University（清华大学）
Completing a corrupted image with correct structures and reasonable textures for a mixed scene remains an elusive challenge. Since the missing hole in a mixed scene of a corrupted image often contains various semantic information, conventional two-stage approaches utilizing structural information often lead to the problem of unreliable structural prediction and ambiguous image texture generation. In this paper, we propose a Semantic Guidance and Evaluation Network (SGE-Net) to iteratively update the structural priors and the inpainted image in an interplay framework of semantics extraction and…
Learning Temporal 3D Semantic Scene Completion via Optical Flow Guidance
College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机科学与电子工程学院）；State Key Laboratory of Advanced Design and Manufacturing Technology for Vehicle, Hunan University（车辆先进设计与制造技术国家重点实验室）
Self-supervised depth estimation has made a great success in learning depth from unlabeled image sequences. While the mappings between image and pixel-wise depth are well-studied in current methods, the correlation between image, depth and scene semantics, however, is less considered. This hinders the network to better understand the real geometry of the scene, since the contextual clues, contribute not only the latent representations of scene depth, but also the straight constraints for depth map. In this paper, we leverage the two benefits by proposing the implicit and explicit semantic guid…
TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；College of Computer Science, Nankai University（南开大学计算机学院）
RRPN is among the outstanding scene text detection approaches, but the manually-designed anchor and coarse proposal refinement make the performance still far from perfection. In this paper, we propose RRPN++ to exploit the potential of RRPN-based model by several improvements. Based on RRPN, we propose the Anchor-free Pyramid Proposal Networks (APPN) to generate first-stage proposals, which adopts the anchor-free design to reduce proposal number and accelerate the inference speed. In our second stage, both the detection branch and the recognition branch are incorporated to perform multi-task l…
One-Shot Crowd Counting With Density Guidance For Scene Adaptation
Jiangsu Key Laboratory of Intelligent Weather Forecasting and Applications Based on Big Data（江苏大数据智能天气预报与应用重点实验室）；State Key Laboratory of Climate System Prediction and Risk Management (CPRM)（气候变化预测与风险管理国家重点实验室）；ICAR/CIC-FEMD/KLME/ILCEC；Nanjing University of Information Science and Technology（南京信息工程大学）；School of Artificial Intelligence, OPtics and ElectroNics, Northwestern Polytechnical University（人工智能、光学与电子学学院，西北工业大学）；School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）
UIUC（伊利诺伊大学厄巴纳-香槟分校）；IBM Research（IBM研究院）；ReLER, UTS（悉尼科技大学ReLER实验室）；University of Oregon（俄勒冈大学）
Multi-scale context module and single-stage encoder-decoder structure are commonly employed for semantic segmentation. The multi-scale context module refers to the operations to aggregate feature responses from a large spatial extent, while the single-stage encoder-decoder structure encodes the high-level semantic information in the encoder path and recovers the boundary information in the decoder path. In contrast, multi-stage encoder-decoder networks have been widely used in human pose estimation and show superior performance than their single-stage counterpart. However, few efforts have bee…
Efficient RGB-D Scene Understanding via Multi-task Adaptive Learning and Cross-dimensional Feature Guidance
School of Mechanical Engineering（机械工程学院）；Vehicle Measurement, Control and Safety Key Laboratory of Sichuan Province（四川省车辆测量控制与安全重点实验室）；National Key Laboratory for Novel Software Technology（新型软件技术国家实验室）；Shanghai Advanced Research Institute（上海先进研究院）；Hubei Key Laboratory of Modern Manufacturing Quality Engineering（湖北省现代制造质量工程重点实验室）
One Model for Two Tasks: Cooperatively Recognizing and Recovering Low-Resolution Scene Text Images by Iterative Mutual Guidance
School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Shanghai Key Lab of Intelligent Information Processing（上海智能信息处理重点实验室）；Tongji University（同济大学）
VOIC: Visible-Occluded Integrated Guidance for 3D Semantic Scene Completion
Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology（可信自主系统研究院，南方科技大学）；Department of Computer Science and Engineering, Southern University of Science and Technology（计算机科学与工程系，南方科技大学）；School of Computer Science, University of Nottingham Ningbo China（宁波大学计算机学院）；Department of Electronic and Information Engineering, Changchun University（电子与信息工程学院，长春大学）
Arbitrary Reading Order Scene Text Spotter with Local Semantics Guidance
Institute of Information Engineering, Chinese Academy of Science（中国科学院信息工程研究所）；Nankai University（南开大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Chinese Academy of Sciences（中国科学院大学）
University of Science and Technology of China（中国科学技术大学）
Scene text removal (STR) contains two processes: text localization and background reconstruction. Through integrating both processes into a single network, previous methods provide an implicit erasure guidance by modifying all pixels in the entire image. However, there exists two problems: 1) the implicit erasure guidance causes the excessive erasure to non-text areas; 2) the one-stage erasure lacks the exhaustive removal of text region. In this paper, we propose a ProgrEssively Region-based scene Text eraser (PERT), introducing an explicit erasure guidance and performing balanced multi-stage …
Diffusion generative models have recently greatly improved the power of text-conditioned image generation. Existing image generation models mainly include text conditional diffusion model and cross-modal guided diffusion model, which are good at small scene image generation and complex scene image generation respectively. In this work, we propose a simple yet effective approach, namely UPainting, to unify simple and complex scene image generation, as shown in Figure 1. Based on architecture improvements and diverse guidance schedules, UPainting effectively integrates cross-modal guidance from …
Zhejiang University of Science & Technology（浙江科技大学）
RGB thermal scene parsing has recently attracted increasing research interest in the field of computer vision. However, most existing methods fail to perform good boundary extraction for prediction maps and cannot fully use high level features. In addition, these methods simply fuse the features from RGB and thermal modalities but are unable to obtain comprehensive fused features. To address these problems, we propose an edge-aware guidance fusion network (EGFNet) for RGB thermal scene parsing. First, we introduce a prior edge map generated using the RGB and thermal images to capture detailed …
Department of Electronics and Computer Engineering, Hanyang University（汉阳大学电子与计算机工程系）
Estimating accurate depth from a single image is challenging because it is an ill-posed problem as infinitely many 3D scenes can be projected to the same 2D scene. However, recent works based on deep convolutional neural networks show great progress with plausible results. The convolutional neural networks are generally composed of two parts: an encoder for dense feature extraction and a decoder for predicting the desired depth. In the encoder-decoder schemes, repeated strided convolution and spatial pooling layers lower the spatial resolution of transitional outputs, and several techniques su…
University of Leeds（利兹大学）；Xi’an Jiaotong University（西安交通大学）
Crowd simulation is a central topic in several fields including graphics. To achieve high-fidelity simulations, data has been increasingly relied upon for analysis and simulation guidance. However, the information in real-world data is often noisy, mixed and unstructured, making it difficult for effective analysis, therefore has not been fully utilized. With the fast-growing volume of crowd data, such a bottleneck needs to be addressed. In this paper, we propose a new framework which comprehensively tackles this problem. It centers at an unsupervised method for analysis. The method takes as in…
Heavy occlusion and dense gathering in crowd scene make pedestrian detection become a challenging problem, because it's difficult to guess a precise full bounding box according to the invisible human part. To crack this nut, we propose a mechanism called Visible Feature Guidance (VFG) for both training and inference. During training, we adopt visible feature to regress the simultaneous outputs of visible bounding box and full bounding box. Then we perform NMS only on visible bounding boxes to achieve the best fitting full box in inference. This manner can alleviate the incapable influence brou…
