Co-occurrent visual pattern makes context aggregation become an essential paradigm for semantic segmentation.The existing studies focus on modeling the contexts within image while neglecting the valuable semantics of the corresponding category beyond image. To this end, we propose a novel soft mining contextual information beyond image paradigm named MCIBI++ to further boost the pixel-level representations. Specifically, we first set up a dynamically updated memory module to store the dataset-level distribution information of various categories and then leverage the information to yield the dataset-level category representations during network forward. After that, we generate a class probability distribution for each pixel representation and conduct the dataset-level context aggregation with the class probability distribution as weights. Finally, the original pixel representations are augmented with the aggregated dataset-level and the conventional image-level contextual information. Moreover, in the inference phase, we additionally design a coarse-to-fine iterative inference strategy to further boost the segmentation results. MCIBI++ can be effortlessly incorporated into the existing segmentation frameworks and bring consistent performance improvements. Also, MCIBI++ can be extended into the video semantic segmentation framework with considerable improvements over the baseline. Equipped with MCIBI++, we achieved the state-of-the-art performance on seven challenging image or video semantic segmentation benchmarks.
University of Science and Technology of China（中国科学技术大学）；ByteDance（字节跳动）
This paper studies the context aggregation problem in semantic image segmentation. The existing researches focus on improving the pixel representations by aggregating the contextual information within individual images. Though impressive, these methods neglect the significance of the representations of the pixels of the corresponding class beyond the input image. To address this, this paper proposes to mine the contextual information beyond individual images to further augment the pixel representations. We first set up a feature memory module, which is updated dynamically during training, to s…
While impressive progress has been achieved, video instance segmentation (VIS) methods with per-clip input often fail on challenging videos with occluded objects and crowded scenes. This is mainly because instance queries in these methods cannot encode well the discriminative embeddings of instances, making the query-based segmenter difficult to distinguish those `hard' instances. To address these issues, we propose to mine discriminative query embeddings (MDQE) to segment occluded instances on challenging videos. First, we initialize the positional embeddings and content features of object qu…
Learning Local and Global Temporal Contexts for Video Semantic Segmentation
ETH Zurich（苏黎世联邦理工学院）；Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局信息通信研究院）；Nanyang Technological University（南洋理工大学）
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；Megvii Research Nanjing, Megvii Technology（旷视科技南京研究院）
State-of-the-art two-stage object detectors apply a classifier to a sparse set of object proposals, relying on region-wise features extracted by RoIPool or RoIAlign as inputs. The region-wise features, in spite of aligning well with the proposal locations, may still lack the crucial context information which is necessary for filtering out noisy background detections, as well as recognizing objects possessing no distinctive appearances. To address this issue, we present a simple but effective Hierarchical Context Embedding (HCE) framework, which can be applied as a plug-and-play component, to f…
Tianjin University（天津大学）；University of Wollongong（伍伦贡大学）
Recent contrastive based 3D action representation learning has made great progress. However, the strict positive/negative constraint is yet to be relaxed and the use of non-self positive is yet to be explored. In this paper, a Contrastive Positive Mining (CPM) framework is proposed for unsupervised skeleton 3D action representation learning. The CPM identifies non-self positives in a contextual queue to boost learning. Specifically, the siamese encoders are adopted and trained to match the similarity distributions of the augmented instances in reference to all instances in the contextual queue…
Object categorization is a hot issue of an image mining. Contextual information between objects is one of the important semantic knowledge of an image. However, the previous researches for an object categorization have not made full use of the contextual information, especially the spatial relations between objects. In addition, the object categorization methods, which generally use the probabilistic graphical models to implement the incorporation of contextual information with appearance of objects, are almost inevitable to evaluate the intractable partition function for normalization. In thi…
A Renaissance of Explicit Motion Information Mining from Transformers for Action Recognition
Shanghai AI Laboratory（上海人工智能实验室）；The University of Sydney（悉尼大学）；SenseTime Group Ltd.（时光机器集团有限公司）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究所，中国科学院）
R2GenCSR: Mining Contextual and Residual Information for LLMs-based Radiology Report Generation
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；First Affiliated Hospital of Anhui University of Chinese Medicine（安徽中医药大学第一附属医院）
National Space Science Center, Chinese Academy of Sciences（中国科学院国家空间科学中心）；University of Chinese Academy of Sciences（中国科学院大学）；University of Rochester（罗切斯特大学）；Meituan（美团）
3D dense captioning, as an emerging vision-language task, aims to identify and locate each object from a set of point clouds and generate a distinctive natural language sentence for describing each located object. However, the existing methods mainly focus on mining inter-object relationship, while ignoring contextual information, especially the non-object details and background environment within the point clouds, thus leading to low-quality descriptions, such as inaccurate relative position information. In this paper, we make the first attempt to utilize the point clouds clustering features …
Web images come in hand with valuable contextual information. Although this information has long been mined for various uses such as image annotation, clustering of images, inference of image semantic content, etc., insufficient attention has been given to address issues in mining this contextual information. In this paper, we propose a webpage segmentation algorithm targeting the extraction of web images and their contextual information based on their characteristics as they appear on webpages. We conducted a user study to obtain a human-labeled dataset to validate the effectiveness of our me…
Huazhong University of Science and Technology（华中科技大学）；Tencent YouTu Lab（腾讯优图实验室）
Recently, the semantics of scene text has been proven to be essential in fine-grained image classification. However, the existing methods mainly exploit the literal meaning of scene text for fine-grained recognition, which might be irrelevant when it is not significantly related to objects/scenes. We propose an end-to-end trainable network that mines implicit contextual knowledge behind scene text image and enhance the semantics and correlation to fine-tune the image representation. Unlike the existing methods, our model integrates three modalities: visual feature extraction, text semantics ex…
Classifying hand-written digits and letters has taken a big leap with the introduction of ConvNets. However, on very constrained hardware the time necessary to train such models would be high. Our main contribution is twofold. First, we extensively test an end-to-end vanilla neural network (MLP) approach in pure numpy without any pre-processing or feature extraction done beforehand. Second, we show that basic data mining operations can significantly improve the performance of the models in terms of computational time, without sacrificing much accuracy. We illustrate our claims on a simpler var…
AgMTR: Agent Mining Transformer for Few-shot Segmentation in Remote Sensing
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Electronics, Chinese Academy of Sciences（中国科学院电子学研究所）；Tsinghua University（清华大学）
PAII Inc.（PAII公司）；Vanderbilt University（范德堡大学）；Virginia Polytechnic Institute and State University（弗吉尼亚理工学院暨州立大学）；Ping An Insurance (Group) Company of China, Ltd.（中国平安保险（集团）股份有限公司）；University of Florida（佛罗里达大学）
Acquiring large-scale medical image data, necessary for training machine learning algorithms, is frequently intractable, due to prohibitive expert-driven annotation costs. Recent datasets extracted from hospital archives, e.g., DeepLesion, have begun to address this problem. However, these are often incompletely or noisily labeled, e.g., DeepLesion leaves over 50% of its lesions unlabeled. Thus, effective methods to harvest missing annotations are critical for continued progress in medical image analysis. This is the goal of our work, where we develop a powerful system to harvest missing lesio…
Hangzhou Dianzi University（杭州电子科技大学）；Australian Artificial Intelligence Institute（澳大利亚人工智能研究所）；University of Technology Sydney（悉尼科技大学）
Cross-view geo-localization is to spot images of the same geographic target from different platforms, e.g., drone-view cameras and satellites. It is challenging in the large visual appearance changes caused by extreme viewpoint variations. Existing methods usually concentrate on mining the fine-grained feature of the geographic target in the image center, but underestimate the contextual information in neighbor areas. In this work, we argue that neighbor areas can be leveraged as auxiliary information, enriching discriminative clues for geolocalization. Specifically, we introduce a simple and …
KU Leuven（鲁汶大学）；SUT（谢里夫理工大学）；UMBC（马里兰大学巴尔的摩县分校）；ETH Zurich（苏黎世联邦理工学院）
The recognition of human actions and the determination of human attributes are two tasks that call for fine-grained classification. Indeed, often rather small and inconspicuous objects and features have to be detected to tell their classes apart. In order to deal with this challenge, we propose a novel convolutional neural network that mines mid-level image patches that are sufficiently dedicated to resolve the corresponding subtleties. In particular, we train a newly de- signed CNN (DeepPattern) that learns discriminative patch groups. There are two innovative aspects to this. On the one hand…
ActionHub: A Large-scale Action Video Description Dataset for Zero-shot Action Recognition
Sun Yat-sen University（中山大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong University of Science and Technology（香港科技大学）；University of California, Santa Cruz（加州大学圣克鲁兹分校）；Peng Cheng Laboratory（鹏城实验室）
GraphMamba: An Efficient Graph Structure Learning Vision Mamba for Hyperspectral Image Classification
College of Electrical and Information Engineering, Hunan University（湖南大学电气与信息工程学院）；Peng Cheng Laboratory（鹏城实验室）；School of Information Engineering, Zhongnan University of Economics and Law（中南财经政法大学信息工程学院）；Emergency Management Research Center, Zhongnan University of Economics and Law（中南财经政法大学应急管理研究中心）
