Tag-based image retrieval (TBIR) has drawn much attention in recent years due to the explosive amount of digital images and crowdsourcing tags. However, the TBIR applications still suffer from the deficient and inaccurate tags provided by users. Inspired by the subspace clustering methods, we formulate the tag completion problem in a subspace clustering model which assumes that images are sampled from subspaces, and complete the tags using the state-of-the-art Low Rank Representation (LRR) method. And we propose a matrix completion algorithm to further refine the tags. Our empirical results on multiple benchmark datasets for image annotation show that the proposed algorithm outperforms state-of-the-art approaches when handling missing and noisy tags.
Soochow University（苏州大学）；Huawei Technologies Co., Ltd.（华为技术有限公司）；Suning Commerce R&D Center USA, Inc.（苏宁美国研发中心）；Travelers（Travelers公司）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Jiangsu University of Technology（江苏理工学院）；Intermediate People’s Court of Linyi City（临沂市中级人民法院）；Hebei University of Economics and Business（河北经贸大学）；Savitribai Phule Pune University（萨维特里巴伊·普勒浦那大学）
In the problems of image retrieval and annotation, complete textual tag lists of images play critical roles. However, in real-world applications, the image tags are usually incomplete, thus it is important to learn the complete tags for images. In this paper, we study the problem of image tag complete and proposed a novel method for this problem based on a popular image representation method, convolutional neural network (CNN). The method estimates the complete tags from the convolutional filtering outputs of images based on a linear predictor. The CNN parameters, linear predictor, and the com…
Renmin University of China（中国人民大学）；University of Florence（佛罗伦萨大学）；Stanford University（斯坦福大学）；University of Amsterdam（阿姆斯特丹大学）；Qualcomm Research Netherlands（荷兰高通研究院）
Where previous reviews on content-based image retrieval emphasize on what can be seen in an image to bridge the semantic gap, this survey considers what people tag about an image. A comprehensive treatise of three closely linked problems, i.e., image tag assignment, refinement, and tag-based image retrieval is presented. While existing works vary in terms of their targeted tasks and methodology, they rely on the key functionality of tag relevance, i.e. estimating the relevance of a specific tag with respect to the visual content of a given image and its social context. By analyzing what inform…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；University of Warwick（华威大学）
The social media explosion has populated the Internet with a wealth of images. There are two existing paradigms for image retrieval: 1) content-based image retrieval (CBIR), which has traditionally used visual features for similarity search (e.g., SIFT features), and 2) tag-based image retrieval (TBIR), which has relied on user tagging (e.g., Flickr tags). CBIR now gains semantic expressiveness by advances in deep-learning-based detection of visual labels. TBIR benefits from query-and-click logs to automatically infer more informative labels. However, learning-based tagging still yields noisy …
Annotating images with tags is useful for indexing and retrieving images. However, many available annotation data include missing or inaccurate annotations. In this paper, we propose an image annotation framework which sequentially performs tag completion and refinement. We utilize the subspace property of data via sparse subspace clustering for tag completion. Then we propose a novel matrix completion model for tag refinement, integrating visual correlation, semantic correlation and the novelly studied property of complex errors. The proposed method outperforms the state-of-the-art approaches…
Graduate University of Chinese Academy of Sciences（中国科学院大学）；Soochow University（苏州大学）；National Time Service Center, Chinese Academy of Sciences（中国科学院国家授时中心）；Lehigh University（理海大学）；Cornell University（康奈尔大学）；Shandong University of Finance and Economics（山东财经大学）；The University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）
The problem of tag completion is to learn the missing tags of an image. In this paper, we propose to learn a tag scoring vector for each image by local linear learning. A local linear function is used in the neighborhood of each image to predict the tag scoring vectors of its neighboring images. We construct a unified objective function for the learning of both tag scoring vectors and local linear function parame- ters. In the objective, we impose the learned tag scoring vectors to be consistent with the known associations to the tags of each image, and also minimize the prediction error of ea…
Renmin University of China（中国人民大学）；Peking University（北京大学）
This paper presents a new framework for visual bag-of-words (BOW) refinement and reduction to overcome the drawbacks associated with the visual BOW model which has been widely used for image classification. Although very influential in the literature, the traditional visual BOW model has two distinct drawbacks. Firstly, for efficiency purposes, the visual vocabulary is commonly constructed by directly clustering the low-level visual feature vectors extracted from local keypoints, without considering the high-level semantics of images. That is, the visual BOW model still suffers from the semant…
Nanjing University of Science and Technology（南京理工大学）；Fudan University（复旦大学）；University of Texas at San Antonio（得克萨斯大学圣安东尼奥分校）
Image retagging aims to improve tag quality of social images by refining their original tags or assigning new high-quality tags. Recent approaches simultaneously explore visual, user and tag information to improve the performance of image retagging by constructing and exploring an image-tag-user graph. However, such methods will become computationally infeasible with the rapidly increasing number of images, tags and users. It has been proven that Anchor Graph Regularization (AGR) can significantly accelerate large-scale graph learning model by exploring only a small number of anchor points. In…
A novel tag completion algorithm is proposed in this paper, which is designed with the following features: 1) Low-rank and error s-parsity: the incomplete initial tagging matrix D is decomposed into the complete tagging matrix A and a sparse error matrix E. However, instead of minimizing its nuclear norm, A is further factor-ized into a basis matrix U and a sparse coefficient matrix V, i.e. D=UV+E. This low-rank formulation encapsulating sparse coding enables our algorithm to recover latent structures from noisy initial data and avoid performing too much denoising; 2) Local reconstruction stru…
Point clouds are often sparse and incomplete. Existing shape completion methods are incapable of generating details of objects or learning the complex point distributions. To this end, we propose a cascaded refinement network together with a coarse-to-fine strategy to synthesize the detailed object shapes. Considering the local details of partial input with the global shape information together, we can preserve the existing details in the incomplete point set and generate the missing parts with high fidelity. We also design a patch discriminator that guarantees every local area has the same pa…
University of Technology Sydney（悉尼科技大学）；Nanjing University of Science and Technology（南京理工大学）；University of Adelaide（阿德莱德大学）
The number of social images has exploded by the wide adoption of social networks, and people like to share their comments about them. These comments can be a description of the image, or some objects, attributes, scenes in it, which are normally used as the user-provided tags. However, it is well-known that user-provided tags are incomplete and imprecise to some extent. Directly using them can damage the performance of related applications, such as the image annotation and retrieval. In this paper, we propose to learn an image annotation model and refine the user-provided tags simultaneously i…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Delft University of Technology（代尔夫特理工大学）；Lingnan University（岭南大学）；The Hong Kong Polytechnic University（香港理工大学）
How will you repair a physical object with some missings? You may imagine its original shape from previously captured images, recover its overall (global) but coarse shape first, and then refine its local details. We are motivated to imitate the physical repair procedure to address point cloud completion. To this end, we propose a cross-modal shape-transfer dual-refinement network (termed CSDN), a coarse-to-fine paradigm with images of full-cycle participation, for quality point cloud completion. CSDN mainly consists of "shape fusion" and "dual-refinement" modules to tackle the cross-modal cha…
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai AI Laboratory（上海人工智能实验室）
Point cloud completion aims to predict complete shape from its partial observation. Current approaches mainly consist of generation and refinement stages in a coarse-to-fine style. However, the generation stage often lacks robustness to tackle different incomplete variations, while the refinement stage blindly recovers point clouds without the semantic awareness. To tackle these challenges, we unify point cloud Completion by a generic Pretrain-Prompt-Predict paradigm, namely CP3. Inspired by prompting approaches from NLP, we creatively reinterpret point cloud generation and refinement as the p…
Enhancing 3D Semantic Scene Completion with a Refinement Module
National Science Center for Earthquake Engineering, Tianjin University（天津大学地球quake工程科学中心）；School of Civil Engineering, Tianjin University（天津大学土木工程学院）；Chair of Robotics, Artificial Intelligence and Real-time Systems, Technical University of Munich（慕尼黑技术大学机器人、人工智能与实时系统教授席）
Università degli Studi di Firenze（佛罗伦萨大学）；Stanford University（斯坦福大学）；Università degli Studi di Modena e Reggio Emilia（摩德纳-雷焦艾米利亚大学）
Tagging of visual content is becoming more and more widespread as web-based services and social networks have popularized tagging functionalities among their users. These user-generated tags are used to ease browsing and exploration of media collections, e.g. using tag clouds, or to retrieve multimedia content. However, not all media are equally tagged by users. Using the current systems is easy to tag a single photo, and even tagging a part of a photo, like a face, has become common in sites like Flickr and Facebook. On the other hand, tagging a video sequence is more complicated and time con…
Universidad Nacional de San Agustín（圣奥古斯丁国立大学）；Department of Computer Science, University of Chile（智利大学计算机科学系）；Universidad La Salle（拉萨尔大学）
Point cloud completion is the task of predicting complete geometry from partial observations using a point set representation for a 3D shape. Previous approaches propose neural networks to directly estimate the whole point cloud through encoder-decoder models fed by the incomplete point set. By predicting the complete model, the current methods compute redundant information because the output also contains the known incomplete input geometry. This paper proposes an end-to-end neural network architecture that focuses on computing the missing geometry and merging the known input and the predicte…
The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）；Nanyang Technological University（南洋理工大学）；University of California, San Diego（加利福尼亚大学圣迭戈分校）
3D point cloud is an important 3D representation for capturing real world 3D objects. However, real-scanned 3D point clouds are often incomplete, and it is important to recover complete point clouds for downstream applications. Most existing point cloud completion methods use Chamfer Distance (CD) loss for training. The CD loss estimates correspondences between two point clouds by searching nearest neighbors, which does not capture the overall point density distribution on the generated shape, and therefore likely leads to non-uniform point cloud generation. To tackle this problem, we propose …
Point clouds are often sparse and incomplete, which imposes difficulties for real-world applications. Existing shape completion methods tend to generate rough shapes without fine-grained details. Considering this, we introduce a two-branch network for shape completion. The first branch is a cascaded shape completion sub-network to synthesize complete objects, where we propose to use the partial input together with the coarse output to preserve the object details during the dense point reconstruction. The second branch is an auto-encoder to reconstruct the original partial input. The two branch…
Discovering automatically the semantic structure of tagged visual data (e.g. web videos and images) is important for visual data analysis and interpretation, enabling the machine intelligence for effectively processing the fast-growing amount of multi-media data. However, this is non-trivial due to the need for jointly learning underlying correlations between heterogeneous visual and tag data. The task is made more challenging by inherently sparse and incomplete tags. In this work, we develop a method for modelling the inherent visual data concept structures based on a novel Hierarchical-Multi…
PCKRF: Point Cloud Completion and Keypoint Refinement With Fusion Data for 6D Pose Estimation
Faculty of Information Technology, Beijing University of Technology（北京工业大学信息学部）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Beijing Institute of Technology（北京理工大学）；Shanghai Jiao Tong University（上海交通大学）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）
Technical University of Munich（慕尼黑工业大学）；Klinikum rechts der Isar（Isar河畔临床中心）；TranslaTUM Center for Translational Cancer Research（TranslaTUM转化癌症研究中心）；Department for Oral & Maxillofacial Surgery, University Hospital Schleswig-Holstein, Campus Kiel（石勒苏益格-荷尔斯泰因大学医院基尔校区口腔颌面外科）
Designing of a cranial implant needs a 3D understanding of the complete skull shape. Thus, taking a 2D approach is sub-optimal, since a 2D model lacks a holistic 3D view of both the defective and healthy skulls. Further, loading the whole 3D skull shapes at its original image resolution is not feasible in commonly available GPUs. To mitigate these issues, we propose a fully convolutional network composed of two subnetworks. The first subnetwork is designed to complete the shape of the downsampled defective skull. The second subnetwork upsamples the reconstructed shape slice-wise. We train the …
We present CIRCLE, a framework for large-scale scene completion and geometric refinement based on local implicit signed distance functions. It is based on an end-to-end sparse convolutional network, CircNet, that jointly models local geometric details and global scene structural contexts, allowing it to preserve fine-grained object detail while recovering missing regions commonly arising in traditional 3D scene data. A novel differentiable rendering module enables test-time refinement for better reconstruction quality. Extensive experiments on both real-world and synthetic datasets show that o…
University of California, Santa Barbara（加州大学圣巴巴拉分校）；Purdue University（普渡大学）
We consider the problem of accurately identifying cell boundaries and labeling individual cells in confocal microscopy images, specifically, 3D image stacks of cells with tagged cell membranes. Precise identification of cell boundaries, their shapes, and quantifying inter-cellular space leads to a better understanding of cell morphogenesis. Towards this, we outline a cell segmentation method that uses a deep neural network architecture to extract a confidence map of cell boundaries, followed by a 3D watershed algorithm and a final refinement using a conditional random field. In addition to imp…
The University of Hong Kong（香港大学）；University of Massachusetts, Amherst（马萨诸塞大学阿默斯特分校）
We propose a data-driven method for recovering miss-ing parts of 3D shapes. Our method is based on a new deep learning architecture consisting of two sub-networks: a global structure inference network and a local geometry refinement network. The global structure inference network incorporates a long short-term memorized context fusion module (LSTM-CF) that infers the global structure of the shape based on multi-view depth information provided as part of the input. It also includes a 3D fully convolutional (3DFCN) module that further enriches the global structure representation according to vol…
S2ML: Spatio-Spectral Mutual Learning for Depth Completion
Institute of Data and Information, Tsinghua Shenzhen International Graduate School, Tsinghua University（数据与信息研究所，清华大学深圳国际研究生院，清华大学）；School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；Department of Electrical Engineering and the Institute of Communications Engineering, National Tsing Hua University（电子工程系和通信工程研究所，国立清华大学）
RailSyn: Diagnosis-Guided Image Generation for Traceable Data Completion in Railway Foreign Object Detection
School of Information Science and Technology, Beijing University of Technology（北京工业大学信息科学与技术学院）；Pratt School of Engineering, Duke University（杜克大学普拉特工程学院）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；School of Mathematical Sciences, University of Chinese Academy of Sciences（中国科学院大学数学科学学院）；Industrial Systems Engineering and Management, National University of Singapore（新加坡国立大学工业系统工程与管理系）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
