Mass flow estimation is of great importance to several industries, and it can be quite challenging to obtain accurate estimates due to limitation in expense or general infeasibility. In the context of agricultural applications, yield monitoring is a key component to precision agriculture and mass flow is the critical factor to measure. Measuring mass flow allows for field productivity analysis, cost minimization, and adjustments to machine efficiency. Methods such as volume or force-impact have been used to measure mass flow; however, these methods are limited in application and accuracy. In this work, we use deep learning to develop and test a vision system that can accurately estimate the mass of sugarcane while running in real-time on a sugarcane harvester during operation. The deep learning algorithm that is used to estimate mass flow is trained using very sparsely annotated images (semi-supervised) using only final load weights (aggregated weights over a certain period of time). The deep neural network (DNN) succeeds in capturing the mass of sugarcane accurately and surpasses older volumetric-based methods, despite highly varying lighting and material colors in the images. The deep neural network is initially trained to predict mass on laboratory data (bamboo) and then transfer learning is utilized to apply the same methods to estimate mass of sugarcane. Using a vision system with a relatively lightweight deep neural network we are able to estimate mass of bamboo with an average error of 4.5% and 5.9% for a select season of sugarcane.
Object detectors usually achieve promising results with the supervision of complete instance annotations. However, their performance is far from satisfactory with sparse instance annotations. Most existing methods for sparsely annotated object detection either re-weight the loss of hard negative samples or convert the unlabeled instances into ignored regions to reduce the interference of false negatives. We argue that these strategies are insufficient since they can at most alleviate the negative effect caused by missing annotations. In this paper, we propose a simple but effective mechanism, …
Boston University（波士顿大学）；University of North Carolina（北卡罗来纳大学）
Our work introduces a novel way to increase pose estimation accuracy by discovering parts from unannotated regions of training images. Discovered parts are used to generate more accurate appearance likelihoods for traditional part-based models like Pictorial Structures [13] and its derivatives. Our experiments on images of a hawkmoth in flight show that our proposed approach significantly improves over existing work [27] for this application, while also being more generally applicable. Our proposed approach localizes landmarks at least twice as accurately as a baseline based on a Mixture of Pi…
Beijing Institute of Technology（北京理工大学）；MEGVII Technology（旷视科技）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）
Sparsely annotated semantic segmentation (SASS) aims to train a segmentation network with coarse-grained (i.e., point-, scribble-, and block-wise) supervisions, where only a small proportion of pixels are labeled in each image. In this paper, we propose a novel tree energy loss for SASS by providing semantic guidance for unlabeled pixels. The tree energy loss represents images as minimum spanning trees to model both low-level and high-level pair-wise affinities. By sequentially applying these affinities to the network prediction, soft pseudo labels for unlabeled pixels are generated in a coars…
European Laboratory for Non-Linear Spectroscopy (LENS), University of Florence（佛罗伦萨大学欧洲非线性光谱实验室）；Bioretics SrL（Bioretics SrL公司）；National Institute of Optics, National Research Council (INO-CNR)（国家研究委员会国家光学研究所）；Department of Physics（物理系）
Semantic segmentation of neuronal structures in 3D high-resolution fluorescence microscopy imaging of the human brain cortex can take advantage of bidimensional CNNs, which yield good results in neuron localization but lead to inaccurate surface reconstruction. 3D CNNs, on the other hand, would require manually annotated volumetric data on a large scale and hence considerable human effort. Semi-supervised alternative strategies which make use only of sparse annotations suffer from longer training times and achieved models tend to have increased capacity compared to 2D CNNs, needing more ground…
Semantic segmentation based on sparse annotation has advanced in recent years. It labels only part of each object in the image, leaving the remainder unlabeled. Most of the existing approaches are time-consuming and often necessitate a multi-stage training strategy. In this work, we propose a simple yet effective sparse annotated semantic segmentation framework based on segformer, dubbed SASFormer, that achieves remarkable performance. Specifically, the framework first generates hierarchical patch attention maps, which are then multiplied by the network predictions to produce correlated region…
Efficient and reliable methods for training of object detectors are in higher demand than ever, and more and more data relevant to the field is becoming available. However, large datasets like Open Images Dataset v4 (OID) are sparsely annotated, and some measure must be taken in order to ensure the training of a reliable detector. In order to take the incompleteness of these datasets into account, one possibility is to use pretrained models to detect the presence of the unverified objects. However, the performance of such a strategy depends largely on the power of the pretrained model. In this…
Although many approaches for multi-human pose estimation in videos have shown profound results, they require densely annotated data which entails excessive man labor. Furthermore, there exists occlusion and motion blur that inevitably lead to poor estimation performance. To address these problems, we propose a method that leverages an attention mask for occluded joints and encodes temporal dependency between frames using transformers. First, our framework composes different combinations of sparsely annotated frames that denote the track of the overall joint movement. We propose an occlusion at…
In training object detector based on convolutional neural networks, selection of effective positive examples for training is an important factor. However, when training an anchor-based detectors with sparse annotations on an image, effort to find effective positive examples can hinder training performance. When using the anchor-based training for the ground truth bounding box to collect positive examples under given IoU, it is often possible to include objects from other classes in the current training class, or objects that are needed to be trained can only be sampled as negative examples. We…
Fully supervised object detection requires training images in which all instances are annotated. This is actually impractical due to the high labor and time costs and the unavoidable missing annotations. As a result, the incomplete annotation in each image could provide misleading supervision and harm the training. Recent works on sparsely annotated object detection alleviate this problem by generating pseudo labels for the missing annotations. Such a mechanism is sensitive to the threshold of the pseudo label score. However, the effective threshold is different in different training stages an…
Pulmonary embolisms (PE) are known to be one of the leading causes for cardiac-related mortality. Due to inherent variabilities in how PE manifests and the cumbersome nature of manual diagnosis, there is growing interest in leveraging AI tools for detecting PE. In this paper, we build a two-stage detection pipeline that is accurate, computationally efficient, robust to variations in PE types and kernels used for CT reconstruction, and most importantly, does not require dense annotations. Given the challenges in acquiring expert annotations in large-scale datasets, our approach produces state-o…
Unified Unsupervised and Sparsely-Supervised 3D Object Detection by Semantic Pseudo-Labeling and Prototype Learning
Institute of Medical Robotics, Shanghai Jiao Tong University（上海交通大学医疗机器人研究院）；Key Laboratory of System Control and Information Processing, Ministry of Education of China（系统控制与信息处理教育部重点实验室）
Saratoga High School（萨拉托加高中）；Stanford University（斯坦福大学）
Chest X-rays of coronavirus disease 2019 (COVID-19) patients are frequently obtained to determine the extent of lung disease and are a valuable source of data for creating artificial intelligence models. Most work to date assessing disease severity on chest imaging has focused on segmenting computed tomography (CT) images; however, given that CTs are performed much less frequently than chest X-rays for COVID-19 patients, automated lung lesion segmentation on chest X-rays could be clinically valuable. There currently exists a universal shortage of chest X-rays with ground truth COVID-19 lung le…
To obtain 3D annotations, we are restricted to controlled environments or synthetic datasets, leading us to 3D datasets with less generalizability to real-world scenarios. To tackle this issue in the context of semi-supervised 3D hand shape and pose estimation, we propose the Pose Alignment network to propagate 3D annotations from labelled frames to nearby unlabelled frames in sparsely annotated videos. We show that incorporating the alignment supervision on pairs of labelled-unlabelled frames allows us to improve the pose estimation accuracy. Besides, we show that the proposed Pose Alignment …
University of Freiburg（弗莱堡大学）；BIOSS Centre for Biological Signalling Studies（BIOSS生物信号研究中心）；University Medical Center Freiburg（弗莱堡大学医学中心）；University Hospital Freiburg（弗莱堡大学医院）；Google DeepMind（谷歌DeepMind）
This paper introduces a network for volumetric segmentation that learns from sparsely annotated volumetric images. We outline two attractive use cases of this method: (1) In a semi-automated setup, the user annotates some slices in the volume to be segmented. The network learns from these sparse annotations and provides a dense 3D segmentation. (2) In a fully-automated setup, we assume that a representative, sparsely annotated training set exists. Trained on this data set, the network densely segments new volumetric images. The proposed network extends the previous u-net architecture from Ronn…
Multi-atlas segmentation is a widely used tool in medical image analysis, providing robust and accurate results by learning from annotated atlas datasets. However, the availability of fully annotated atlas images for training is limited due to the time required for the labelling task. Segmentation methods requiring only a proportion of each atlas image to be labelled could therefore reduce the workload on expert raters tasked with annotating atlas images. To address this issue, we first re-examine the labelling problem common in many existing approaches and formulate its solution in terms of a…
JD AI Research（京东人工智能研究院）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；University of Science and Technology of China (USTC)（中国科学技术大学）
Existing person re-identification (re-id) methods are stuck when deployed to a new unseen scenario despite the success in cross-camera person matching. Recent efforts have been substantially devoted to domain adaptive person re-id where extensive unlabeled data in the new scenario are utilized in a transductive learning manner. However, for each scenario, it is required to first collect enough data and then train such a domain adaptive re-id model, thus restricting their practical application. Instead, we aim to explore multiple labeled datasets to learn generalized domain-invariant representa…
SP3D: Boosting Sparsely-Supervised 3D Object Detection via Accurate Cross-Modal Semantic Prompts
Fujian Key Laboratory of Sensing and Computing for Smart Cities, Xiamen University（厦门大学福建省智慧城市感知与计算重点实验室）；Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（厦门大学多媒体可信感知与高效计算教育部重点实验室）
Yale University（耶鲁大学）；IBM Almaden Research Center（IBM阿尔马登研究中心）；New York University（纽约大学）
Pulmonary Embolism (PE) is a life-threatening disorder associated with high mortality and morbidity. Prompt diagnosis and immediate initiation of therapeutic action is important. We explored a deep learning model to detect PE on volumetric contrast-enhanced chest CT scans using a 2-stage training strategy. First, a residual convolutional neural network (ResNet) was trained using annotated 2D images. In addition to the classification loss, an attention loss was added during training to help the network focus attention on PE. Next, a recurrent network was used to scan sequentially through the fe…
Frequency Prior Guided Matching: A Data Augmentation Approach for Generalizable Semi-Supervised Polyp Segmentation
Academy of Medical Engineering and Translational Medicine, Tianjin University（天津大学医学工程与转化医学研究院）；Hangzhou Institute of Medicine, Chinese Academy of Sciences（杭州医学研究所）；Yale University（耶鲁大学）
Imperial College London（伦敦帝国理工学院）；Technical University Munich（慕尼黑工业大学）
Deep neural networks exhibit limited generalizability across images with different entangled domain features and categorical features. Learning generalizable features that can form universal categorical decision boundaries across domains is an interesting and difficult challenge. This problem occurs frequently in medical imaging applications when attempts are made to deploy and improve deep learning models across different image acquisition devices, across acquisition parameters or if some classes are unavailable in new training databases. To address this problem, we propose Mutual Information…
Carnegie Mellon University（卡内基梅隆大学）；University of Pittsburgh（匹兹堡大学）
Vessel segmentation is an essential task in many clinical applications. Although supervised methods have achieved state-of-art performance, acquiring expert annotation is laborious and mostly limited for two-dimensional datasets with a small sample size. On the contrary, unsupervised methods rely on handcrafted features to detect tube-like structures such as vessels. However, those methods require complex pipelines involving several hyper-parameters and design choices rendering the procedure sensitive, dataset-specific, and not generalizable. We propose a self-supervised method with a limited …
Are Dense Labels Always Necessary for 3D Object Detection from Point Cloud?
School of Intelligent Systems Engineering, the Shenzhen Campus of Sun Yatsen University（南方科技大学深圳校区智能系统工程学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Mathematics and Statistics and Ministry of Education Key Lab of Intelligent Networks and Network Security, Xi’an Jiaotong University（西安交通大学数学与统计学系和教育部智能网络与网络安全重点实验室）；School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications（重庆邮电大学通信与信息工程学院）；Meta；University of Maryland（美国马里兰大学）
Sun Yat-sen University（中山大学）；DarkMatter AI Research（暗物质人工智能研究院）；Shenzhen Research Institute of Big Data, the Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）深圳大数据研究院）；Megvii Technology（旷视科技）
Deep learning-based video salient object detection has recently achieved great success with its performance significantly outperforming any other unsupervised methods. However, existing data-driven approaches heavily rely on a large quantity of pixel-wise annotated video frames to deliver such promising results. In this paper, we address the semi-supervised video salient object detection task using pseudo-labels. Specifically, we present an effective video saliency detector that consists of a spatial refinement network and a spatiotemporal module. Based on the same refinement network and motio…
EndoMatcher: Generalizable Endoscopic Image Matcher via Multi-Domain Pre-training for Robot-Assisted Surgery
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）；Department of Pulmonary and Critical Care Medicine, The First Affiliated Hospital of Sun Yat-sen University（中山大学附属第一医院呼吸与危重症医学科）；Hong Kong Institute of Science & Innovation（香港创新科技署）；Centre of AI and Robotics, Hong Kong Institute of Science & Innovation（人工智能与机器人中心，香港创新科技署）
Hong Kong University of Science and Technology（香港科技大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Kwai Inc.（快手公司）
Partially supervised instance segmentation aims to perform learning on limited mask-annotated categories of data thus eliminating expensive and exhaustive mask annotation. The learned models are expected to be generalizable to novel categories. Existing methods either learn a transfer function from detection to segmentation, or cluster shape priors for segmenting novel categories. We propose to learn the underlying class-agnostic commonalities that can be generalized from mask-annotated categories to novel categories. Specifically, we parse two types of commonalities: 1) shape commonalities wh…
3D Foundation Model for Generalizable Disease Detection in Head Computed Tomography
New York University, Center for Data Science（纽约大学数据科学中心）；New York University, Courant Institute of Mathematical Sciences（纽约大学Courant数学科学研究所）；NYU Grossman School of Medicine, Department of Radiology（纽约大学 Grossman 医学院放射科）；Siemens Healthineers（西门子医疗科技）；NYU Grossman School of Medicine, Department of Neurology（纽约大学 Grossman 医学院神经病学部）；NYU Grossman School of Medicine, Department of Neuroscience and Physiology（纽约大学 Grossman 医学院神经科学与生理学部）；NYU Grossman School of Medicine, Neuroscience Institute（纽约大学 Grossman 医学院神经科学研究所）
A multimodal vision foundation model for generalizable knee pathology
Beihang University（北京航空航天大学）；Peking University Third Hospital（北京大学第三医院）；Institute of Sports Medicine of Peking University（北京大学运动医学研究所）；Ministry of Education（教育部）
Recent trends in semi-supervised learning have significantly boosted the performance of 3D semi-supervised medical image segmentation. Compared with 2D images, 3D medical volumes involve information from different directions, e.g., transverse, sagittal, and coronal planes, so as to naturally provide complementary views. These complementary views and the intrinsic similarity among adjacent 3D slices inspire us to develop a novel annotation way and its corresponding semi-supervised model for effective segmentation. Specifically, we firstly propose the orthogonal annotation by only labeling two o…
Brown University（布朗大学）；Preferred Networks, Inc.（优选网络公司）；Toyota Research Institute (TRI)（丰田研究院）
Deep learning models for semantic segmentation rely on expensive, large-scale, manually annotated datasets. Labelling is a tedious process that can take hours per image. Automatically annotating video sequences by propagating sparsely labeled frames through time is a more scalable alternative. In this work, we propose a novel label propagation method, termed Warp-Refine Propagation, that combines semantic cues with geometric cues to efficiently auto-label videos. Our method learns to refine geometrically-warped labels and infuse them with learned semantic priors in a semi-supervised setting by…
University of California San Diego（加州大学圣迭戈分校）；Cornell University（康奈尔大学）；Cornell Tech（康奈尔科技学院）
Given the vast amounts of video available online, and recent breakthroughs in object detection with static images, object detection in video offers a promising new frontier. However, motion blur and compression artifacts cause substantial frame-level variability, even in videos that appear smooth to the eye. Additionally, video datasets tend to have sparsely annotated frames. We present a new framework for improving object detection in videos that captures temporal context and encourages consistency of predictions. First, we train a pseudo-labeler, that is, a domain-adapted convolutional neura…
ARTORG Center for Biomedical Engineering Research, University of Bern（伯尔尼大学ARTORG生物医学工程研究中心）；Department of Emergency Medicine, Bern University Hospital “Inselspital”（伯尔尼大学医院“Inselspital”急诊医学系）；University Clinic for Pneumonology, Bern University Hospital “Inselspital”（伯尔尼大学医院“Inselspital”肺科大学诊所）；Department of Diagnostic, Interventional and Pediatric Radiology, Bern University Hospital “Inselspital”（伯尔尼大学医院“Inselspital”诊断、介入和儿科放射学系）；Bern University Hospital “Inselspital”（伯尔尼大学医院“Inselspital”）；University of Bern（伯尔尼大学）
Early and accurate diagnosis of interstitial lung diseases (ILDs) is crucial for making treatment decisions, but can be challenging even for experienced radiologists. The diagnostic procedure is based on the detection and recognition of the different ILD pathologies in thoracic CT scans, yet their manifestation often appears similar. In this study, we propose the use of a deep purely convolutional neural network for the semantic segmentation of ILD patterns, as the basic component of a computer aided diagnosis (CAD) system for ILDs. The proposed CNN, which consists of convolutional layers with…
