Mask is All You Need: Rethinking Mask R-CNN for Dense and Arbitrary-Shaped Scene Text Detection
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Guangzhou University（广州大学）；Mashang Consumer Finance Co., Ltd.（马上消费金融股份有限公司）
Xugong Qin, Yu Zhou, Youhui Guo, Dayan Wu, Zhihong Tian, Ning Jiang, Hongbin Wang, Weiping Wang
Due to the large success in object detection and instance segmentation, Mask R-CNN attracts great attention and is widely adopted as a strong baseline for arbitrary-shaped scene text detection and spotting. However, two issues remain to be settled. The first is dense text case, which is easy to be neglected but quite practical. There may exist multiple instances in one proposal, which makes it difficult for the mask head to distinguish different instances and degrades the performance. In this work, we argue that the performance degradation results from the learning confusion issue in the mask head. We propose to use an MLP decoder instead of the "deconv-conv" decoder in the mask head, which alleviates the issue and promotes robustness significantly. And we propose instance-aware mask learning in which the mask head learns to predict the shape of the whole instance rather than classify each pixel to text or non-text. With instance-aware mask learning, the mask branch can learn separated and compact masks. The second is that due to large variations in scale and aspect ratio, RPN needs complicated anchor settings, making it hard to maintain and transfer across different datasets. To settle this issue, we propose an adaptive label assignment in which all instances especially those with extreme aspect ratios are guaranteed to be associated with enough anchors. Equipped with these components, the proposed method named MAYOR achieves state-of-the-art performance on five benchmarks including DAST1500, MSRA-TD500, ICDAR2015, CTW1500, and Total-Text.
IP-SAM: Rethinking Prompt-Conditioned Segmentation for Prompt-Absent Deployment
University of Chinese Academy of Sciences（中国科学院大学）；Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences（中国科学院空间应用工程与技术中心）
We report competitive results on object detection and instance segmentation on the COCO dataset using standard models trained from random initialization. The results are no worse than their ImageNet pre-training counterparts even when using the hyper-parameters of the baseline system (Mask R-CNN) that were optimized for fine-tuning pre-trained models, with the sole exception of increasing the number of training iterations so the randomly initialized models may converge. Training from random initialization is surprisingly robust; our results hold even when: (i) using only 10% of the training da…
Institute of Technology for Nanostructures (NST)（纳米结构技术研究所（NST））；Center for Nanointegration Duisburg-Essen (CENIDE)（杜伊斯堡-埃森纳米集成中心（CENIDE））；University of Duisburg-Essen（杜伊斯堡-埃森大学）
Fiber-shaped materials (e.g. carbon nano tubes) are of great relevance, due to their unique properties but also the health risk they can impose. Unfortunately, image-based analysis of fibers still involves manual annotation, which is a time-consuming and costly process. We therefore propose the use of region-based convolutional neural networks (R-CNNs) to automate this task. Mask R-CNN, the most widely used R-CNN for semantic segmentation tasks, is prone to errors when it comes to the analysis of fiber-shaped objects. Hence, a new architecture - FibeR-CNN - is introduced and validated. FibeR-C…
SAMSOD: Rethinking SAM Optimization for RGB-T Salient Object Detection
Key Laboratory of Intelligent Computing and Signal Processing of Ministry of Education, School of Computer Science and Technology, Anhui University（教育部智能计算与信号处理重点实验室，计算机科学与技术学院，安徽大学）
FZI Research Center for Information Technology（FZI信息技术研究中心）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院（KIT））
Recent advances in the area of plane segmentation from single RGB images show strong accuracy improvements and now allow a reliable segmentation of indoor scenes into planes. Nonetheless, fine-grained details of these segmentation masks are still lacking accuracy, thus restricting the usability of such techniques on a larger scale in numerous applications, such as inpainting for Augmented Reality use cases. We propose a post-processing algorithm to align the segmented plane masks with edges detected in the image. This allows us to increase the accuracy of state-of-the-art approaches, while lim…
ETH Zurich（苏黎世联邦理工学院）；University of Technology Sydney（悉尼科技大学）
Prevalent semantic segmentation solutions, despite their different network designs (FCN based or attention based) and mask decoding strategies (parametric softmax based or pixel-query based), can be placed in one category, by considering the softmax weights or query vectors as learnable class prototypes. In light of this prototype view, this study uncovers several limitations of such parametric segmentation regime, and proposes a nonparametric alternative based on non-learnable prototypes. Instead of prior methods learning a single weight/query vector for each class in a fully parametric manne…
Massachusetts Institute of Technology（麻省理工学院）；Skoltech（斯科尔科沃科学技术研究院）
In design of instance segmentation networks that reconstruct masks, segmentation is often taken as its literal definition -- assigning each pixel a label. This has led to thinking the problem as a template matching one with the goal of minimizing the loss between the reconstructed and the ground truth pixels. Rethinking reconstruction networks as a generator, we define the problem of predicting masks as a GANs game framework: A segmentation network generates the masks, and a discriminator network decides on the quality of the masks. To demonstrate this game, we show effective modifications on …
VPNeXt -- Rethinking Dense Decoding for Plain Vision Transformer
School of Information and Software Engineering, University of Electronic Science and Technology of China（信息与软件工程学院，电子科学与技术大学）；Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China（深圳先进研究所，电子科学与技术大学）
Northeastern University（东北大学）；Key Laboratory of Intelligent Computing in Medical Image of Ministry of Education, Northeastern University（东北大学医学影像智能计算教育部重点实验室）；Amii, University of Alberta（阿尔伯塔大学阿尔伯塔机器智能研究所）
Most recent semantic segmentation methods adopt a U-Net framework with an encoder-decoder architecture. It is still challenging for U-Net with a simple skip connection scheme to model the global multi-scale context: 1) Not each skip connection setting is effective due to the issue of incompatible feature sets of encoder and decoder stage, even some skip connection negatively influence the segmentation performance; 2) The original U-Net is worse than the one without any skip connection on some datasets. Based on our findings, we propose a new segmentation framework, named UCTransNet (with a pro…
Rethinking Transparent Object Grasping: Depth Completion with Monocular Depth Estimation and Instance Mask
State Key Laboratory of Robotics and System at Harbin Institute of Technology（哈尔滨工业大学机器人系统国家重点实验室）；Lanzhou University of Technology（兰州理工大学）；Department of Computer Science（计算机科学系）；University of Liverpool（利物浦大学）
University of Technology Sydney（悉尼科技大学）；Zhejiang University（浙江大学）
Recently, remarkable progress has been made in weakly supervised object localization (WSOL) to promote object localization maps. The common practice of evaluating these maps applies an indirect and coarse way, i.e., obtaining tight bounding boxes which can cover high-activation regions and calculating intersection-over-union (IoU) scores between the predicted and ground-truth boxes. This measurement can evaluate the ability of localization maps to some extent, but we argue that the maps should be measured directly and delicately, i.e., comparing the maps with the ground-truth object masks pixe…
Rethinking Boundary Detection in Deep Learning-Based Medical Image Segmentation
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（计算机科学与工程系，香港科学与技术大学）；Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology（电子与计算机工程系，香港科学与技术大学）；Department of Chemical and Biological Engineering, The Hong Kong University of Science and Technology（化学与生物工程系，香港科学与技术大学）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深圳-香港协同创新研究院）
Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token
Beijing Institute of Technology（北京理工大学）；University of Birmingham（伯明翰大学）；Beijing Jiaotong University（北京交通大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
Recently fast arbitrary-shaped text detection has become an attractive research topic. However, most existing methods are non-real-time, which may fall short in intelligent systems. Although a few real-time text methods are proposed, the detection accuracy is far behind non-real-time methods. To improve the detection accuracy and speed simultaneously, we propose a novel fast and accurate text detection framework, namely CM-Net, which is constructed based on a new text representation method and a multi-perspective feature (MPF) module. The former can fit arbitrary-shaped text contours by concen…
China University of Mining and Technology（中国矿业大学）；Engineering Research Center of Mine Digitization, Ministry of Education of the People’s Republic of China（中华人民共和国教育部矿山数字化工程研究中心）
Arbitrary-shaped scene text detection is a challenging task due to the variety of text changes in font, size, color, and orientation. Most existing regression based methods resort to regress the masks or contour points of text regions to model the text instances. However, regressing the complete masks requires high training complexity, and contour points are not sufficient to capture the details of highly curved texts. To tackle the above limitations, we propose a novel light-weight anchor-free text detection framework called TextDCT, which adopts the discrete cosine transform (DCT) to encode …
National Institute of Informatics（国立信息学研究所）；University of Electronic Science and Technology of China（电子科技大学）；University of Tokyo（东京大学）
We have investigated a new application of adversarial examples, namely location privacy protection against landmark recognition systems. We introduce mask-guided multimodal projected gradient descent (MM-PGD), in which adversarial examples are trained on different deep models. Image contents are protected by analyzing the properties of regions to identify the ones most suitable for blending in adversarial examples. We investigated two region identification strategies: class activation map-based MM-PGD, in which the internal behaviors of trained deep models are targeted; and human-vision-based …
UIUC（伊利诺伊大学厄巴纳-香槟分校）；Adobe（奥多比）；University of Oregon（俄勒冈大学）
Text segmentation is a prerequisite in many real-world text-related tasks, e.g., text style transfer, and scene text removal. However, facing the lack of high-quality datasets and dedicated investigations, this critical prerequisite has been left as an assumption in many works, and has been largely overlooked by current research. To bridge this gap, we proposed TextSeg, a large-scale fine-annotated text dataset with six types of annotations: word- and character-wise bounding polygons, masks and transcriptions. We also introduce Text Refinement Network (TexRNet), a novel text segmentation appro…
South China University of Technology（华南理工大学）；SenseTime Research（商汤科技研究院）；Pazhou Lab（琶洲实验室）；Shanghai AI Laboratory（上海人工智能实验室）
One of the main challenges for arbitrary-shaped text detection is to design a good text instance representation that allows networks to learn diverse text geometry variances. Most of existing methods model text instances in image spatial domain via masks or contour point sequences in the Cartesian or the polar coordinate system. However, the mask representation might lead to expensive post-processing, while the point sequence one may have limited capability to model texts with highly-curved shapes. To tackle these problems, we model text instances in the Fourier domain and propose one novel Fo…
Reading text from natural images is challenging due to the great variety in text font, color, size, complex background and etc.. The perspective distortion and non-linear spatial arrangement of characters make it further difficult. While rectification based method is intuitively grounded and has pushed the envelope by far, its potential is far from being well exploited. In this paper, we present a bag of tricks that prove to significantly improve the performance of rectification based method. On curved text dataset, our method achieves an accuracy of 89.6% on CUTE-80 and 76.3% on Total-Text, a…
Peking University（北京大学）；University of Chinese Academy of Sciences（中国科学院大学）；Advanced Institute of Information Technology, Peking University（北京大学前沿信息研究院）；University of International Business and Economics（对外经济贸易大学）；Research Center on Fictitious Economy and Data Science, Chinese Academy of Sciences（中国科学院虚拟经济与数据科学研究中心）
Pixel-level road crack detection has always been a challenging task in intelligent transportation systems. Due to the external environments, such as weather, light, and other factors, pavement cracks often present low contrast, poor continuity, and different sizes in length and width. However, most of the existing studies pay less attention to crack data under different situations. Meanwhile, recent algorithms based on deep convolutional neural networks (DCNNs) have promoted the development of cutting-edge models for crack detection. Nevertheless, they usually focus on complex models for good …
MEVL-STP: Multi-Encoder and Vision Language Model for Arbitrarily Shaped Scene Text Spotting
Rajiv Gandhi Institute of Petroleum Technology（拉吉夫·甘地石油技术学院）；Indian Institute of Technology (ISM) Dhanbad（印度理工学院（ISM）丹巴德分校）；University of Salford（索尔福德大学）
Shanghai Institute for Advanced Communication and Data Science（上海先进通信与数据科学研究院）；Shanghai University（上海大学）
Text detection/localization, as an important task in computer vision, has witnessed substantialadvancements in methodology and performance with convolutional neural networks. However, the vastmajority of popular methods use rectangles or quadrangles to describe text regions. These representationshave inherent drawbacks, especially relating to dense adjacent text and loose regional text boundaries,which usually cause difficulty detecting arbitrarily shaped text. In this paper, we propose a novel text regionrepresentation method, with a robust pipeline, which can precisely detect dense adjacent …
DiC: Rethinking Conv3x3 Designs in Diffusion Models
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University（通用人工智能国家重点实验室，智能科学与技术学院，北京大学）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（人工智能学院，北京邮电大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）；School of Mathematical Sciences, Peking University（数学科学学院，北京大学）
Princeton University（普林斯顿大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
We introduce Mask-ToF, a method to reduce flying pixels (FP) in time-of-flight (ToF) depth captures. FPs are pervasive artifacts which occur around depth edges, where light paths from both an object and its background are integrated over the aperture. This light mixes at a sensor pixel to produce erroneous depth estimates, which can adversely affect downstream 3D vision tasks. Mask-ToF starts at the source of these FPs, learning a microlens-level occlusion mask which effectively creates a custom-shaped sub-aperture for each sensor pixel. This modulates the selection of foreground and backgroun…
