PASTA通过自监督Vision Transformer特征空间分布分析，利用Segment Anything Model 3的语义文本提示，实现弱图像监督下的目标和异常分割，提升工业和农业领域的实时处理与分割精度。
摘要：
在无结构环境中检测未见过的异常是一个对工业和农业应用如材料回收和除草至关重要的挑战。现有感知系统由于依赖于详尽标注的数据集，难以满足这些领域对实时处理、像素级分割精度和鲁棒准确性的严格要求。为了解决这些限制，我们提出了一种名为'目标和异常分割的补丁聚合'（PASTA）的弱监督管道，通过将观察场景与名义参考进行比较，在自监督Vision Transformer（ViT）特征空间中进行分布分析，以识别目标和异常对象。我们的管道利用Segment Anything Model 3的语义文本提示进行零样本对象分割。在定制的钢铁废料回收数据集和植物数据集上的评估显示，我们的方法将训练时间减少到领域特定基线的75.8%。尽管是领域无关的，我们的方法在工业和农业领域实现了优越的目标（最高88.3% IoU）和异常（最高63.5% IoU）分割性能。
英文摘要：
Detecting unseen anomalies in unstructured environments presents a critical challenge for industrial and agricultural applications such as material recycling and weeding. Existing perception systems frequently fail to satisfy the strict operational requirements of these domains, specifically real-time processing, pixel-level segmentation precision, and robust accuracy, due to their reliance on exhaustively annotated datasets. To address these limitations, we propose a weakly supervised pipeline for object segmentation and classification using weak image-level supervision called 'Patch Aggregation for Segmentation of Targets and Anomalies' (PASTA). By comparing an observed scene with a nominal reference, PASTA identifies Target and Anomaly objects through distribution analysis in self-supervised Vision Transformer (ViT) feature spaces. Our pipeline utilizes semantic text-prompts via the Segment Anything Model 3 to guide zero-shot object segmentation.
Evaluations on a custom steel scrap recycling dataset and a plant dataset demonstrate a 75.8% training time reduction of our approach to domain-specific baselines. While being domain-agnostic, our method achieves superior Target (up to 88.3% IoU) and Anomaly (up to 63.5% IoU) segmentation performance in the industrial and agricultural domain.
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Carnegie Mellon University（卡内基梅隆大学）；UiT The Arctic University of Norway（挪威北极大学）；ByteDance（字节跳动）
Image-based virtual try-on is one of the most promising applications of human-centric image generation due to its tremendous real-world potential. In this work, we take a step forwards to explore versatile virtual try-on solutions, which we argue should possess three main properties, namely, they should support unsupervised training, arbitrary garment categories, and controllable garment editing. To this end, we propose a characteristic-preserving end-to-end network, the PAtch-routed SpaTially-Adaptive GAN++ (PASTA-GAN++), to achieve a versatile system for high-resolution unpaired virtual try-…
Stochastic-based Patch Filtering for Few-Shot Learning
AIBA, Departament de Matemàtiques & Informàtica Universitat de Barcelona（AIBA，数学与信息学院巴塞罗那大学）；Departamento de Ingeniería de Sistemas y Computación Universidad Católica del Norte（系统与计算工程系天主教北方大学）；Institute of Neuroscience, Universitat de Barcelona（神经科学研究所巴塞罗那大学）
University of Technology Sydney（悉尼科技大学）；Griffith University（格里菲斯大学）
Scene parsing from images is a fundamental yet challenging problem in visual content understanding. In this dense prediction task, the parsing model assigns every pixel to a categorical label, which requires the contextual information of adjacent image patches. So the challenge for this learning task is to simultaneously describe the geometric and semantic properties of objects or a scene. In this paper, we explore the effective use of multi-layer feature outputs of the deep parsing networks for spatial-semantic consistency by designing a novel feature aggregation module to generate the approp…
Shenzhen Campus of Sun Yat-Sen University（中山大学深圳校区）；UiT The Arctic University of Norway（挪威北极大学）；Peng Cheng Laboratory（鹏城实验室）
Image-based virtual try-on is one of the most promising applications of human-centric image generation due to its tremendous real-world potential. Yet, as most try-on approaches fit in-shop garments onto a target person, they require the laborious and restrictive construction of a paired training dataset, severely limiting their scalability. While a few recent works attempt to transfer garments directly from one person to another, alleviating the need to collect paired datasets, their performance is impacted by the lack of paired (supervised) information. In particular, disentangling style and…
Mix-Domain Contrastive Learning for Unpaired H&E-to-IHC Stain Translation
Toronto Metropolitan University（多伦多都会大学）；Zhengzhou University（郑州大学）；The First Affiliated Hospital of Zhengzhou University（郑州大学第一附属医院）；Zhongyuan University of Technology（中原工学院）
University of Texas at Arlington（得克萨斯大学阿灵顿分校）；University of New South Wales（新南威尔士大学）；UNSW Sydney（新南威尔士大学悉尼分校）
Traditional image-based survival prediction models rely on discriminative patch labeling which make those methods not scalable to extend to large datasets. Recent studies have shown Multiple Instance Learning (MIL) framework is useful for histopathological images when no annotations are available in classification task. Different to the current image-based survival models that limit to key patches or clusters derived from Whole Slide Images (WSIs), we propose Deep Attention Multiple Instance Survival Learning (DeepAttnMISL) by introducing both siamese MI-FCN and attention-based MIL pooling to …
Convolutional Neural Network (CNN) models have become the state-of-the-art for most computer vision tasks with natural images. However, these are not best suited for multi-gigapixel resolution Whole Slide Images (WSIs) of histology slides due to large size of these images. Current approaches construct smaller patches from WSIs which results in the loss of contextual information. We propose to capture the spatial context using novel Representation-Aggregation Network (RAN) for segmentation purposes, wherein the first network learns patch-level representation and the second network aggregates co…
Meta AI；Sorbonne University（索邦大学）；Inria（法国国家信息与自动化研究所）
We show how to augment any convolutional network with an attention-based global map to achieve non-local reasoning. We replace the final average pooling by an attention-based aggregation layer akin to a single transformer block, that weights how the patches are involved in the classification decision. We plug this learned aggregation layer with a simplistic patch-based convolutional network parametrized by 2 parameters (width and depth). In contrast with a pyramidal design, this architecture family maintains the input patch resolution across all the layers. It yields surprisingly competitive t…
Université Paris Saclay（巴黎萨克雷大学）；Sorbonne Université（索邦大学）；Emlyon（里昂商学院）；Univ Paris Est Creteil（巴黎东部克雷泰伊大学）
In this paper, we propose an approach for Facial Expressions Recognition (FER) based on a deep multi-facial patches aggregation network. Deep features are learned from facial patches using deep sub-networks and aggregated within one deep architecture for expression classification . Several problems may affect the performance of deep-learning based FER approaches, in particular, the small size of existing FER datasets which might not be sufficient to train large deep learning networks. Moreover, it is extremely time-consuming to collect and annotate a large number of facial images. To account f…
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
Images captured nowadays are of varying dimensions with smartphones and DSLR's allowing users to choose from a list of available image resolutions. It is therefore imperative for forensic algorithms such as resampling detection to scale well for images of varying dimensions. However, in our experiments, we observed that many state-of-the-art forensic algorithms are sensitive to image size and their performance quickly degenerates when operated on images of diverse dimensions despite re-training them using multiple image sizes. To handle this issue, we propose a novel pooling strategy called IT…
Nagoya University（名古屋大学）；Nara Institute of Science and Technology（奈良先端科学技术大学院大学）；National Institute of Informatics（国立信息学研究所）；Keio University School of Medicine（庆应义塾大学医学部）；Juntendo University（顺天堂大学）
This paper proposes a segmentation method of infection regions in the lung from CT volumes of COVID-19 patients. COVID-19 spread worldwide, causing many infected patients and deaths. CT image-based diagnosis of COVID-19 can provide quick and accurate diagnosis results. An automated segmentation method of infection regions in the lung provides a quantitative criterion for diagnosis. Previous methods employ whole 2D image or 3D volume-based processes. Infection regions have a considerable variation in their sizes. Such processes easily miss small infection regions. Patch-based process is effecti…
Towards Implicit Aggregation: Robust Image Representation for Place Recognition in the Transformer Era
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）；Pengcheng Laboratory（鹏城实验室）；Shenyang Institute of Automation, Chinese Academy of Sciences（沈阳自动化研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Pazhou Laboratory (Huangpu)（琶洲实验室（黄埔））
Bag of Image Patch Embedding Behind the Success of Self-Supervised Learning
Meta AI - FAIR（Meta人工智能基础研究实验室）；Center for Data Science, New York University（纽约大学数据科学中心）；Inria（法国国家信息与自动化研究所）；École normale supérieure（巴黎高等师范学院）；CNRS（法国国家科学研究中心）；PSL Research University（巴黎文理研究大学）；Redwood Center（红杉中心）；UC Berkeley（加州大学伯克利分校）；Courant Institute（柯朗研究所）
GNCAF: A GNN-based Neighboring Context Aggregation Framework for Tertiary Lymphoid Structures Semantic Segmentation in WSI
CASMI, Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；Clinical Oncology School of Fujian Medical University, Fujian Cancer Hospital（福建医科大学临床医学学院、福建肿瘤医院）；School of Internet, Anhui University（安徽大学互联网学院）
针对 WSI 中 TLS 区域与成熟阶段的端到端语义分割，提出 GNCAF 聚合多跳邻域上下文并结合自注意力指导分割，显著提升 mF1 与 mIoU。
Most of the existing semantic segmentation approaches with image-level class labels as supervision, highly rely on the initial class activation map (CAM) generated from the standard classification network. In this paper, a novel "Progressive Patch Learning" approach is proposed to improve the local details extraction of the classification, producing the CAM better covering the whole object rather than only the most discriminative regions as in CAMs obtained in conventional classification models. "Patch Learning" destructs the feature maps into patches and independently processes each local pat…
Better than Average: Spatially-Aware Aggregation of Segmentation Uncertainty Improves Downstream Performance
Max-Delbrück-Center (MDC)（马克斯·德尔布吕克中心）；Helmholtz Imaging（亥姆霍兹成像）；Charité Universitätsmedizin（柏林夏里特医学院）；Humboldt-Universität zu Berlin（柏林洪堡大学）；University of Potsdam（波茨坦大学）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；Heidelberg University（海德堡大学）
HIPA: Hierarchical Patch Transformer for Single Image Super Resolution
Ocean University of China（中国海洋大学）；University of Manitoba（曼尼托巴大学）；Harbin Institute of Technology（哈尔滨工业大学）；The Hong Kong Polytechnic University（香港理工大学）；University of Alberta（阿尔伯塔大学）；University of Science and Technology of China（中国科学技术大学）
The University of Queensland（昆士兰大学）；CSIRO（澳大利亚联邦科学与工业研究组织）；The University of Newcastle（纽卡斯尔大学）
Direct automatic segmentation of objects from 3D medical imaging, such as magnetic resonance (MR) imaging, is challenging as it often involves accurately identifying a number of individual objects with complex geometries within a large volume under investigation. To address these challenges, most deep learning approaches typically enhance their learning capability by substantially increasing the complexity or the number of trainable parameters within their models. Consequently, these models generally require long inference time on standard workstations operating clinical MR systems and are res…
Representation Transfer of Foundation Models for Ultra-Widefield Retinal Imaging
Institute of Ophthalmology, University College London（伦敦大学学院眼科研究所）；Wake Forest University School of Medicine（维克森林大学医学院）；Virginia Tech-Wake Forest University School of Biomedical Engineering and Sciences（弗吉尼亚理工大学-维克森林大学生物医学工程与科学学院）；University of Zagreb Faculty of Electrical Engineering and Computing（萨格勒布大学电气工程与计算机学院）；NIHR Biomedical Research Centre, Moorfields Eye Hospital NHS Foundation Trust（NIHR生物医学研究中心，摩尔菲尔兹眼科医院NHS基金会信托）
Weakly Supervised Contrastive Learning for Histopathology Patch Embeddings
Department of Electrical and Computer Engineering, University of Utah（电气与计算机工程系，犹他大学）；Scientific Computing and Imaging Institute, University of Utah（科学计算与成像研究所，犹他大学）；Department of Pathology, University of Utah（病理学系，犹他大学）
Validation of Whole-Slide Foundation Models for Image Retrieval in TCGA Data
KIMIA Lab, Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN, USA（KIMIA实验室，人工智能与信息学，梅奥诊所，罗切斯特，明尼苏达州，美国）；Department of Neurology, Northwestern University Feinberg School of Medicine, Chicago, IL, USA（神经病学系，北western大学费因伯格医学院，芝加哥，伊利诺伊州，美国）；Department of Computer Science, Temple University, Philadelphia, PA, USA（计算机科学系，泰勒大学，费城，宾夕法尼亚州，美国）；Department of Breast and Melanoma Surgical Oncology, Comprehensive Cancer Center, Mayo Clinic, Rochester, MN, USA（乳腺和黑色素瘤外科肿瘤学系，综合癌症中心，梅奥诊所，罗切斯特，明尼苏达州，美国）；Department of Oncology, Comprehensive Cancer Center, Mayo Clinic, Rochester, MN, USA（肿瘤学系，综合癌症中心，梅奥诊所，罗切斯特，明尼苏达州，美国）；Department of Quantitative Health Sciences, Mayo Clinic, Rochester, MN, USA（定量健康科学系，梅奥诊所，罗切斯特，明尼苏达州，美国）
Brain-Inspired Stepwise Patch Merging for Vision Transformers
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所脑启发认知人工智能实验室）；State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology（脑认知与脑启发智能技术国家重点实验室）；School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）；Center for Long-term AI（长期人工智能中心）
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所）
Visual place recognition is a challenging task for applications such as autonomous driving navigation and mobile robot localization. Distracting elements presenting in complex scenes often lead to deviations in the perception of visual place. To address this problem, it is crucial to integrate information from only task-relevant regions into image representations. In this paper, we introduce a novel holistic place recognition model, TransVPR, based on vision Transformers. It benefits from the desirable property of the self-attention operation in Transformers which can naturally aggregate task-…
Chongqing University（重庆大学）；San Diego State University（圣地亚哥州立大学）
Automatic pavement distress classification facilitates improving the efficiency of pavement maintenance and reducing the cost of labor and resources. A recently influential branch of this task divides the pavement image into patches and addresses these issues from the perspective of multi-instance learning. However, these methods neglect the correlation between patches and suffer from a low efficiency in the model optimization and inference. Meanwhile, Swin Transformer is able to address both of these issues with its unique strengths. Built upon Swin Transformer, we present a vision Transforme…
Incorporating various mass shapes and sizes in training deep learning architectures has made breast mass segmentation challenging. Moreover, manual segmentation of masses of irregular shapes is time-consuming and error-prone. Though Deep Neural Network has shown outstanding performance in breast mass segmentation, it fails in segmenting micro-masses. In this paper, we propose a novel U-net-shaped transformer-based architecture, called Swin-SFTNet, that outperforms state-of-the-art architectures in breast mammography-based micro-mass segmentation. Firstly to capture the global context, we desig…
University of Science and Technology of China（中国科学技术大学）；Harbin Institute of Technology（哈尔滨工业大学）；SenseTime Research（商汤科技研究院）
Point clouds and RGB images are two general perceptional sources in autonomous driving. The former can provide accurate localization of objects, and the latter is denser and richer in semantic information. Recently, AutoAlign presents a learnable paradigm in combining these two modalities for 3D object detection. However, it suffers from high computational cost introduced by the global-wise attention. To solve the problem, we propose Cross-Domain DeformCAFA module in this work. It attends to sparse learnable sampling points for cross-modal relational modeling, which enhances the tolerance to c…
Thermal Topology Collapse: Universal Physical Patch Attacks on Infrared Vision Systems
China University of Petroleum-Beijing at Karamay（中国石油大学（北京）克拉玛依校区）；University of Electronic Science and Technology of China（电子科技大学）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；School of Software, Shandong University（山东大学软件学院）；Shanghai Jiaotong University（上海交通大学）
Efficient KernelSHAP Explanations for Patch-based 3D Medical Image Segmentation
Department of Biomedical Sciences, Humanitas University（生物医学科学系，人道大学）；Radiotherapy and Radiosurgery Department, IRCCS Humanitas Research Hospital（放射治疗与放射外科系，IRCCS人道研究医院）
The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Sun Yat-sen University（中山大学）；The University of Hong Kong（香港大学）
In recent years, the Transformer architecture has shown its superiority in the video-based person re-identification task. Inspired by video representation learning, these methods mainly focus on designing modules to extract informative spatial and temporal features. However, they are still limited in extracting local attributes and global identity information, which are critical for the person re-identification task. In this paper, we propose a novel Multi-Stage Spatial-Temporal Aggregation Transformer (MSTAT) with two novel designed proxy embedding modules to address the above issue. Specific…
Segmenting salient objects in an image is an important vision task with ubiquitous applications. The problem becomes more challenging in the presence of a cluttered and textured background, low resolution and/or low contrast images. Even though existing algorithms perform well in segmenting most of the object(s) of interest, they often end up segmenting false positives due to resembling salient objects in the background. In this work, we tackle this problem by iteratively attending to image patches in a recurrent fashion and subsequently enhancing the predicted segmentation mask. Saliency feat…
National University of Singapore（新加坡国立大学）；South China University of Technology（华南理工大学）；ByteDance Inc.（字节跳动公司）
Recent Vision Transformer~(ViT) models have demonstrated encouraging results across various computer vision tasks, thanks to their competence in modeling long-range dependencies of image patches or tokens via self-attention. These models, however, usually designate the similar receptive fields of each token feature within each layer. Such a constraint inevitably limits the ability of each self-attention layer in capturing multi-scale features, thereby leading to performance degradation in handling images with multiple objects of different scales. To address this issue, we propose a novel and g…
Visual appearance is considered to be the most important cue to understand images for cross-modal retrieval, while sometimes the scene text appearing in images can provide valuable information to understand the visual semantics. Most of existing cross-modal retrieval approaches ignore the usage of scene text information and directly adding this information may lead to performance degradation in scene text free scenarios. To address this issue, we propose a full transformer architecture to unify these cross-modal retrieval scenarios in a single $\textbf{Vi}$sion and $\textbf{S}$cene $\textbf{T}…
