Semantic human matting aims to estimate the per-pixel opacity of the foreground human regions. It is quite challenging and usually requires user interactive trimaps and plenty of high quality annotated data. Annotating such kind of data is labor intensive and requires great skills beyond normal users, especially considering the very detailed hair part of humans. In contrast, coarse annotated human dataset is much easier to acquire and collect from the public dataset. In this paper, we propose to use coarse annotated data coupled with fine annotated data to boost end-to-end semantic human matting without trimaps as extra input. Specifically, we train a mask prediction network to estimate the coarse semantic mask using the hybrid data, and then propose a quality unification network to unify the quality of the previous coarse mask outputs. A matting refinement network takes in the unified mask and the input image to predict the final alpha matte. The collected coarse annotated dataset enriches our dataset significantly, allows generating high quality alpha matte for real images. Experimental results show that the proposed method performs comparably against state-of-the-art methods. Moreover, the proposed method can be used for refining coarse annotated public dataset, as well as semantic segmentation methods, which reduces the cost of annotating high quality human data to a great extent.
In this paper, we propose an image matting framework called Salient Image Matting to estimate the per-pixel opacity value of the most salient foreground in an image. To deal with a large amount of semantic diversity in images, a trimap is conventionally required as it provides important guidance about object semantics to the matting process. However, creating a good trimap is often expensive and timeconsuming. The SIM framework simultaneously deals with the challenge of learning a wide range of semantics and salient object types in a fully automatic and an end to end manner. Specifically, our …
The University of Adelaide（阿德莱德大学）；Adobe Inc.（Adobe公司）；Zhejiang University（浙江大学）
Deep image matting methods have achieved increasingly better results on benchmarks (e.g., Composition-1k/alphamatting.com). However, the robustness, including robustness to trimaps and generalization to images from different domains, is still under-explored. Although some works propose to either refine the trimaps or adapt the algorithms to real-world images via extra data augmentation, none of them has taken both into consideration, not to mention the significant performance deterioration on benchmarks while using those data augmentation. To fill this gap, we propose an image matting method w…
Cutting out and object and estimate its transparency mask is a key task in many applications. We take on the work on closed-form matting by Levin et al., that is used at the core of many matting techniques, and propose an alternative formulation that offers more flexible controls over the matting priors. We also show that this new approach is efficient at upscaling transparency maps from coarse estimates.
The Johns Hopkins University（约翰霍普金斯大学）；Adobe（Adobe公司）
We propose Mask Guided (MG) Matting, a robust matting framework that takes a general coarse mask as guidance. MG Matting leverages a network (PRN) design which encourages the matting model to provide self-guidance to progressively refine the uncertain regions through the decoding process. A series of guidance mask perturbation operations are also introduced in the training to further enhance its robustness to external guidance. We show that PRN can generalize to unseen types of guidance masks such as trimap and low-quality alpha matte, making it suitable for various application pipelines. In a…
Human matting, high quality extraction of humans from natural images, is crucial for a wide variety of applications. Since the matting problem is severely under-constrained, most previous methods require user interactions to take user designated trimaps or scribbles as constraints. This user-in-the-loop nature makes them difficult to be applied to large scale data or time-sensitive scenarios. In this paper, instead of using explicit user input constraints, we employ implicit semantic constraints learned from data and propose an automatic human matting algorithm (SHM). SHM is the first algorith…
Image matting refers to the estimation of the opacity of foreground objects. It requires correct contours and fine details of foreground objects for the matting results. To better accomplish human image matting tasks, we propose the Cascade Image Matting Network with Deformable Graph Refinement, which can automatically predict precise alpha mattes from single human images without any additional inputs. We adopt a network cascade architecture to perform matting from low-to-high resolution, which corresponds to coarse-to-fine optimization. We also introduce the Deformable Graph Refinement (DGR) …
University of Pennsylvania（宾夕法尼亚大学）；Tsinghua University（清华大学）
We propose a light-weight video frame interpolation algorithm. Our key innovation is an instance-level supervision that allows information to be learned from the high-resolution version of similar objects. Our experiment shows that the proposed method can generate state-of-the-art results across different datasets, with fractional computation resources (time and memory) of competing methods. Given two image frames, a cascade network creates an intermediate frame with 1) a flow-warping module that computes coarse bi-directional optical flow and creates an interpolated image via flow-based warpi…
The University of Hong Kong（香港大学）；University of Oxford（牛津大学）
This paper addresses the problem of image matting for transparent objects. Existing approaches often require tedious capturing procedures and long processing time, which limit their practical use. In this paper, we formulate transparent object matting as a refractive flow estimation problem, and propose a deep learning framework, called TOM-Net, for learning the refractive flow. Our framework comprises two parts, namely a multi-scale encoder-decoder network for producing a coarse prediction, and a residual network for refinement. At test time, TOM-Net takes a single image as input, and outputs…
This paper addresses the problem of transparent object matting. Existing image matting approaches for transparent objects often require tedious capturing procedures and long processing time, which limit their practical use. In this paper, we first formulate transparent object matting as a refractive flow estimation problem. We then propose a deep learning framework, called TOM-Net, for learning the refractive flow. Our framework comprises two parts, namely a multi-scale encoder-decoder network for producing a coarse prediction, and a residual network for refinement. At test time, TOM-Net takes…
ModiFace Inc.（ModiFace 公司）；University of Toronto（多伦多大学）
Augmented reality is an emerging technology in many application domains. Among them is the beauty industry, where live virtual try-on of beauty products is of great importance. In this paper, we address the problem of live hair color augmentation. To achieve this goal, hair needs to be segmented quickly and accurately. We show how a modified MobileNet CNN architecture can be used to segment the hair in real-time. Instead of training this network using large amounts of accurate segmentation data, which is difficult to obtain, we use crowd sourced hair segmentation data. While such data is much …
Group-On: Boosting One-Shot Segmentation with Supportive Query
Zhejiang University（浙江大学）；University of Notre Dame（诺丁汉大学）；Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence（浙江省医学影像人工智能重点实验室）；HKUST(Guangzhou)（香港科技大学（广州））
Boosting Single-domain Generalized Object Detection via Vision-Language Knowledge Interaction
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（先进交叉学科研究院，中国科学院大学）；Institute of Microelectronics of the Chinese Academy of Sciences（中国科学院微电子研究所）
Key Laboratory of Intelligent Information Processing, Advanced Computer Research Center, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所智能信息处理重点实验室（先进计算机研究中心））；School of Computer and Control Engineering, University of Chinese Academy of Sciences (UCAS)（中国科学院大学计算机与控制工程学院）；Department of Radiology, Suining Central Hospital（遂宁市中心医院放射科）；Peking University（北京大学）
Many recently developed object detectors focused on coarse-to-fine framework which contains several stages that classify and regress proposals from coarse-grain to fine-grain, and obtains more accurate detection gradually. Multi-resolution models such as Feature Pyramid Network(FPN) integrate information of different levels of resolution and effectively improve the performance. Previous researches also have revealed that localization can be further improved by: 1) using fine-grained information which is more translational variant; 2) refining local areas which is more focused on local boundary…
AD-Det: Boosting Object Detection in UAV Images with Focused Small Objects and Balanced Tail Classes
College of Computer and Data Science（计算机与数据科学学院）；Engineering Research Center of Big Data Intelligence, Ministry of Education（大数据智能工程研究中心）；Key Laboratory of Intelligent Metro of Universities in Fujian（福建省高校智能地铁重点实验室）；School of Software and Information Engineering（软件与信息工程学院）
Boosting Salient Object Detection with Transformer-based Asymmetric Bilateral U-Net
Nankai University（南开大学）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）；Institute for Infocomm Research (I2R)（资讯通信研究院）；University of Electronic Science and Technology of China (UESTC)（电子科技大学）
University of Technology Sydney（悉尼科技大学）；Tencent AI Lab（腾讯人工智能实验室）；Fujian Normal University（福建师范大学）；University of Nottingham Ningbo China（宁波诺丁汉大学）
Recent segmentation methods, such as OCR and CPNet, utilizing "class level" information in addition to pixel features, have achieved notable success for boosting the accuracy of existing network modules. However, the extracted class-level information was simply concatenated to pixel features, without explicitly being exploited for better pixel representation learning. Moreover, these approaches learn soft class centers based on coarse mask prediction, which is prone to error accumulation. In this paper, aiming to use class level information more effectively, we propose a universal Class-Aware …
Boosting Fine-Grained Visual Anomaly Detection with Coarse-Knowledge-Aware Adversarial Learning
Sun Yat-sen University（中山大学）；Guangdong Key Laboratory of Big Data Analysis and Processing（广东省大数据分析与处理重点实验室）；The Hong Kong Polytechnic University（香港理工大学）；Pazhou Lab（琶洲实验室）
RealignDiff: Boosting Text-to-Image Diffusion Model with Coarse-to-fine Semantic Re-alignment
Pengcheng Laboratory（鹏城实验室）；Sun Yat-sen University（中山大学）；Australian Artificial Intelligence Institute, University of Technology Sydney（悉尼科技大学澳大利亚人工智能研究所）；College of Information Technology (CIT), United Arab Emirates University (UAEU)（阿联酋大学信息技术学院）
Yahoo Labs（雅虎研究院）；National Institutes of Health（美国国立卫生研究院）；Georgia State University（佐治亚州立大学）；Siemens Medical Solutions Inc（西门子医疗解决方案公司）
Classification is one of the core problems in Computer-Aided Diagnosis (CAD), targeting for early cancer detection using 3D medical imaging interpretation. High detection sensitivity with desirably low false positive (FP) rate is critical for a CAD system to be accepted as a valuable or even indispensable tool in radiologists' workflow. Given various spurious imagery noises which cause observation uncertainties, this remains a very challenging task. In this paper, we propose a novel, two-tiered coarse-to-fine (CTF) classification cascade framework to tackle this problem. We first obtain classi…
EyecareGPT: Boosting Comprehensive Ophthalmology Understanding with Tailored Dataset, Benchmark and Model
Zhejiang University（浙江大学）；Harbin Institute of Technology（哈尔滨工业大学）；The First People’s Hospital of Chenzhou（郴州市第一人民医院）；National University of Singapore（新加坡国立大学）
BooM-VVT: Boosting Mask-Free Video Virtual Try-On with Image-Level Pseudo Data
Nanjing University of Science and Technology（南京理工大学）；University of Science and Technology of China（中国科学技术大学）；National University of Singapore（新加坡国立大学）；Meituan（美团）
