Images captured nowadays are of varying dimensions with smartphones and DSLR's allowing users to choose from a list of available image resolutions. It is therefore imperative for forensic algorithms such as resampling detection to scale well for images of varying dimensions. However, in our experiments, we observed that many state-of-the-art forensic algorithms are sensitive to image size and their performance quickly degenerates when operated on images of diverse dimensions despite re-training them using multiple image sizes. To handle this issue, we propose a novel pooling strategy called ITERATIVE POOLING. This pooling strategy can dynamically adjust input tensors in a discrete without much loss of information as in ROI Max-pooling. This pooling strategy can be used with any of the existing deep models and for demonstration purposes, we show its utility on Resnet-18 for the case of resampling detection a fundamental operation for any image sought of image manipulation. Compared to existing strategies and Max-pooling it gives up to 7-8% improvement on public datasets.
Mayachitra Inc.（Mayachitra公司）；University of California, Riverside（加州大学河滨分校）；Naval Air Warfare Center Weapons Division（海军航空作战中心武器部）；University of California, Santa Barbara（加州大学圣巴巴拉分校）
Resampling is an important signature of manipulated images. In this paper, we propose two methods to detect and localize image manipulations based on a combination of resampling features and deep learning. In the first method, the Radon transform of resampling features are computed on overlapping image patches. Deep learning classifiers and a Gaussian conditional random field model are then used to create a heatmap. Tampered regions are located using a Random Walker segmentation method. In the second method, resampling features computed on overlapping image patches are passed through a Long sh…
Tsinghua University（清华大学）；University of Rochester（罗切斯特大学）
Reference-based super-resolution (RefSR) has made significant progress in producing realistic textures using an external reference (Ref) image. However, existing RefSR methods obtain high-quality correspondence matchings consuming quadratic computation resources with respect to the input size, limiting its application. Moreover, these approaches usually suffer from scale misalignments between the low-resolution (LR) image and Ref image. In this paper, we propose an Accelerated Multi-Scale Aggregation network (AMSA) for Reference-based Super-Resolution, including Coarse-to-Fine Embedded PatchMa…
Convolutional Neural Network (CNN) models have become the state-of-the-art for most computer vision tasks with natural images. However, these are not best suited for multi-gigapixel resolution Whole Slide Images (WSIs) of histology slides due to large size of these images. Current approaches construct smaller patches from WSIs which results in the loss of contextual information. We propose to capture the spatial context using novel Representation-Aggregation Network (RAN) for segmentation purposes, wherein the first network learns patch-level representation and the second network aggregates co…
Meta AI；Sorbonne University（索邦大学）；Inria（法国国家信息与自动化研究所）
We show how to augment any convolutional network with an attention-based global map to achieve non-local reasoning. We replace the final average pooling by an attention-based aggregation layer akin to a single transformer block, that weights how the patches are involved in the classification decision. We plug this learned aggregation layer with a simplistic patch-based convolutional network parametrized by 2 parameters (width and depth). In contrast with a pyramidal design, this architecture family maintains the input patch resolution across all the layers. It yields surprisingly competitive t…
Image search systems based on local descriptors typically achieve orientation invariance by aligning the patches on their dominant orientations. Albeit successful, this choice introduces too much invariance because it does not guarantee that the patches are rotated consistently. This paper introduces an aggregation strategy of local descriptors that achieves this covariance property by jointly encoding the angle in the aggregation stage in a continuous manner. It is combined with an efficient monomial embedding to provide a codebook-free method to aggregate local descriptors into a single vect…
Towards Implicit Aggregation: Robust Image Representation for Place Recognition in the Transformer Era
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）；Pengcheng Laboratory（鹏城实验室）；Shenyang Institute of Automation, Chinese Academy of Sciences（沈阳自动化研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Pazhou Laboratory (Huangpu)（琶洲实验室（黄埔））
Emotional Intelligence in Human-Computer Interaction has attracted increasing attention from researchers in multidisciplinary research fields including psychology, computer vision, neuroscience, artificial intelligence, and related disciplines. Human prone to naturally interact with computers face-to-face. Human Expressions is an important key to better link human and computers. Thus, designing interfaces able to understand human expressions and emotions can improve Human-Computer Interaction (HCI) for better communication. In this paper, we investigate HCI via a deep multi-facial patches aggr…
Université Paris Saclay（巴黎萨克雷大学）；Sorbonne Université（索邦大学）；Emlyon（里昂商学院）；Univ Paris Est Creteil（巴黎东部克雷泰伊大学）
In this paper, we propose an approach for Facial Expressions Recognition (FER) based on a deep multi-facial patches aggregation network. Deep features are learned from facial patches using deep sub-networks and aggregated within one deep architecture for expression classification . Several problems may affect the performance of deep-learning based FER approaches, in particular, the small size of existing FER datasets which might not be sufficient to train large deep learning networks. Moreover, it is extremely time-consuming to collect and annotate a large number of facial images. To account f…
An adversarial patch can arbitrarily manipulate image pixels within a restricted region to induce model misclassification. The threat of this localized attack has gained significant attention because the adversary can mount a physically-realizable attack by attaching patches to the victim object. Recent provably robust defenses generally follow the PatchGuard framework by using CNNs with small receptive fields and secure feature aggregation for robust model predictions. In this paper, we extend PatchGuard to PatchGuard++ for provably detecting the adversarial patch attack to boost both provabl…
Rethinking the Readout: Unlocking Video Backbones for AI-Generated Video Detection
Huazhong University of Science and Technology（华中科技大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Jilin University（吉林大学）；Vast Intelligence Lab（旷视智能实验室）
The Chinese University of Hong Kong（香港中文大学）；Kuaishou Technology（快手科技）；Tsinghua University（清华大学）；SmartMore（思谋科技）
Video super-resolution (VSR) aims to utilize multiple low-resolution frames to generate a high-resolution prediction for each frame. In this process, inter- and intra-frames are the key sources for exploiting temporal and spatial information. However, there are a couple of limitations for existing VSR methods. First, optical flow is often used to establish temporal correspondence. But flow estimation itself is error-prone and affects recovery results. Second, similar patterns existing in natural images are rarely exploited for the VSR task. Motivated by these findings, we propose a temporal mu…
GLIMPSE: Global-Local Indexing with Multi-Precision Spectral Encoding for Explorable Video World Models
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Cardiff University（卡迪夫大学）；Kling AI；The Hong Kong University of Science and Technology（香港科技大学）
University of Technology Sydney（悉尼科技大学）；Griffith University（格里菲斯大学）
Scene parsing from images is a fundamental yet challenging problem in visual content understanding. In this dense prediction task, the parsing model assigns every pixel to a categorical label, which requires the contextual information of adjacent image patches. So the challenge for this learning task is to simultaneously describe the geometric and semantic properties of objects or a scene. In this paper, we explore the effective use of multi-layer feature outputs of the deep parsing networks for spatial-semantic consistency by designing a novel feature aggregation module to generate the approp…
Most of the existing semantic segmentation approaches with image-level class labels as supervision, highly rely on the initial class activation map (CAM) generated from the standard classification network. In this paper, a novel "Progressive Patch Learning" approach is proposed to improve the local details extraction of the classification, producing the CAM better covering the whole object rather than only the most discriminative regions as in CAMs obtained in conventional classification models. "Patch Learning" destructs the feature maps into patches and independently processes each local pat…
Nagoya University（名古屋大学）；Nara Institute of Science and Technology（奈良先端科学技术大学院大学）；National Institute of Informatics（国立信息学研究所）；Keio University School of Medicine（庆应义塾大学医学部）；Juntendo University（顺天堂大学）
This paper proposes a segmentation method of infection regions in the lung from CT volumes of COVID-19 patients. COVID-19 spread worldwide, causing many infected patients and deaths. CT image-based diagnosis of COVID-19 can provide quick and accurate diagnosis results. An automated segmentation method of infection regions in the lung provides a quantitative criterion for diagnosis. Previous methods employ whole 2D image or 3D volume-based processes. Infection regions have a considerable variation in their sizes. Such processes easily miss small infection regions. Patch-based process is effecti…
Queensland University of Technology（昆士兰科技大学）；QUT Centre for Robotics（QUT机器人中心）
Visual Place Recognition is a challenging task for robotics and autonomous systems, which must deal with the twin problems of appearance and viewpoint change in an always changing world. This paper introduces Patch-NetVLAD, which provides a novel formulation for combining the advantages of both local and global descriptor methods by deriving patch-level features from NetVLAD residuals. Unlike the fixed spatial neighborhood regime of existing local keypoint features, our method enables aggregation and matching of deep-learned local features defined over the feature-space grid. We further introd…
HIPA: Hierarchical Patch Transformer for Single Image Super Resolution
Ocean University of China（中国海洋大学）；University of Manitoba（曼尼托巴大学）；Harbin Institute of Technology（哈尔滨工业大学）；The Hong Kong Polytechnic University（香港理工大学）；University of Alberta（阿尔伯塔大学）；University of Science and Technology of China（中国科学技术大学）
GNCAF: A GNN-based Neighboring Context Aggregation Framework for Tertiary Lymphoid Structures Semantic Segmentation in WSI
CASMI, Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；Clinical Oncology School of Fujian Medical University, Fujian Cancer Hospital（福建医科大学临床医学学院、福建肿瘤医院）；School of Internet, Anhui University（安徽大学互联网学院）
针对 WSI 中 TLS 区域与成熟阶段的端到端语义分割，提出 GNCAF 聚合多跳邻域上下文并结合自注意力指导分割，显著提升 mF1 与 mIoU。
Bosch Research North America（博世北美研究院）；University of Missouri（密苏里大学）
A well-known challenge in applying deep-learning methods to omnidirectional images is spherical distortion. In dense regression tasks such as depth estimation, where structural details are required, using a vanilla CNN layer on the distorted 360 image results in undesired information loss. In this paper, we propose a 360 monocular depth estimation pipeline, OmniFusion, to tackle the spherical distortion issue. Our pipeline transforms a 360 image into less-distorted perspective patches (i.e. tangent images) to obtain patch-wise predictions via CNN, and then merge the patch-wise results for fina…
SUNY at Buffalo（布法罗纽约州立大学）；Tianjin University（天津大学）
Deep convolutional neural networks (CNN) have recently been shown to generate promising results for aesthetics assessment. However, the performance of these deep CNN methods is often compromised by the constraint that the neural network only takes the fixed-size input. To accommodate this requirement, input images need to be transformed via cropping, warping, or padding, which often alter image composition, reduce image resolution, or cause image distortion. Thus the aesthetics of the original images is impaired because of potential loss of fine grained details and holistic image layout. Howev…
Global operations, such as global average pooling, are widely used in top-performance image restorers. They aggregate global information from input features along entire spatial dimensions but behave differently during training and inference in image restoration tasks: they are based on different regions, namely the cropped patches (from images) and the full-resolution images. This paper revisits global information aggregation and finds that the image-based features during inference have a different distribution than the patch-based features during training. This train-test inconsistency negat…
Nanjing University（南京大学）；Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）
The prediction of microsatellite instability (MSI) and microsatellite stability (MSS) is essential in predicting both the treatment response and prognosis of gastrointestinal cancer. In clinical practice, a universal MSI testing is recommended, but the accessibility of such a test is limited. Thus, a more cost-efficient and broadly accessible tool is desired to cover the traditionally untested patients. In the past few years, deep-learning-based algorithms have been proposed to predict MSI directly from haematoxylin and eosin (H&E)-stained whole-slide images (WSIs). Such algorithms can be summ…
ASUS Intelligent Cloud Services（华硕智能云服务）；Georgia Institute of Technology（佐治亚理工学院）
Single image dehazing is the ill-posed two-dimensional signal reconstruction problem. Recently, deep convolutional neural networks (CNN) have been successfully used in many computer vision problems. In this paper, we propose a Y-net that is named for its structure. This network reconstructs clear images by aggregating multi-scale features maps. Additionally, we propose a Wavelet Structure SIMilarity (W-SSIM) loss function in the training step. In the proposed loss function, discrete wavelet transforms are applied repeatedly to divide the image into differently sized patches with different freq…
Weakly Supervised Contrastive Learning for Histopathology Patch Embeddings
Department of Electrical and Computer Engineering, University of Utah（电气与计算机工程系，犹他大学）；Scientific Computing and Imaging Institute, University of Utah（科学计算与成像研究所，犹他大学）；Department of Pathology, University of Utah（病理学系，犹他大学）
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所）
Visual place recognition is a challenging task for applications such as autonomous driving navigation and mobile robot localization. Distracting elements presenting in complex scenes often lead to deviations in the perception of visual place. To address this problem, it is crucial to integrate information from only task-relevant regions into image representations. In this paper, we introduce a novel holistic place recognition model, TransVPR, based on vision Transformers. It benefits from the desirable property of the self-attention operation in Transformers which can naturally aggregate task-…
University of Science and Technology of China（中国科学技术大学）；Harbin Institute of Technology（哈尔滨工业大学）；SenseTime Research（商汤科技研究院）
Point clouds and RGB images are two general perceptional sources in autonomous driving. The former can provide accurate localization of objects, and the latter is denser and richer in semantic information. Recently, AutoAlign presents a learnable paradigm in combining these two modalities for 3D object detection. However, it suffers from high computational cost introduced by the global-wise attention. To solve the problem, we propose Cross-Domain DeformCAFA module in this work. It attends to sparse learnable sampling points for cross-modal relational modeling, which enhances the tolerance to c…
Dalian Maritime University（大连海事大学）；Dalian Neusoft University of Information（大连东软信息学院）；Memorial University of Newfoundland（纽芬兰纪念大学）；Dalian University（大连大学）
This paper presents an effective normal estimation method adopting multi-patch stitching for an unstructured point cloud. The majority of learning-based approaches encode a local patch around each point of a whole model and estimate the normals in a point-by-point manner. In contrast, we suggest a more efficient pipeline, in which we introduce a patch-level normal estimation architecture to process a series of overlapping patches. Additionally, a multi-normal selection method based on weights, dubbed as multi-patch stitching, integrates the normals from the overlapping patches. To reduce the a…
City University of Hong Kong（香港城市大学）；Flatiron Institute of the Simons Foundation（西蒙斯基金会熨斗研究所）；New York University（纽约大学）
Objective measures of image quality generally operate by comparing pixels of a "degraded" image to those of the original. Relative to human observers, these measures are overly sensitive to resampling of texture regions (e.g., replacing one patch of grass with another). Here, we develop the first full-reference image quality model with explicit tolerance to texture resampling. Using a convolutional neural network, we construct an injective and differentiable function that transforms images to multi-scale overcomplete representations. We demonstrate empirically that the spatial averages of the …
This paper presents a patch-wise low-rank based image denoising method with constrained variational model involving local and nonlocal regularization. On one hand, recent patch-wise methods can be represented as a low-rank matrix approximation problem whose convex relaxation usually depends on nuclear norm minimization (NNM). Here, we extend the NNM to the nonconvex schatten p-norm minimization with additional weights assigned to different singular values, which is referred to as the Weighted Schatten p-Norm Minimization (WSNM). An efficient algorithm is also proposed to solve the WSNM problem…
In recent years, the availability of digitized Whole Slide Images (WSIs) has enabled the use of deep learning-based computer vision techniques for automated disease diagnosis. However, WSIs present unique computational and algorithmic challenges. WSIs are gigapixel-sized ($\sim$100K pixels), making them infeasible to be used directly for training deep neural networks. Also, often only slide-level labels are available for training as detailed annotations are tedious and can be time-consuming for experts. Approaches using multiple-instance learning (MIL) frameworks have been shown to overcome th…
The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Sun Yat-sen University（中山大学）；The University of Hong Kong（香港大学）
In recent years, the Transformer architecture has shown its superiority in the video-based person re-identification task. Inspired by video representation learning, these methods mainly focus on designing modules to extract informative spatial and temporal features. However, they are still limited in extracting local attributes and global identity information, which are critical for the person re-identification task. In this paper, we propose a novel Multi-Stage Spatial-Temporal Aggregation Transformer (MSTAT) with two novel designed proxy embedding modules to address the above issue. Specific…
The University of Queensland（昆士兰大学）；CSIRO（澳大利亚联邦科学与工业研究组织）；The University of Newcastle（纽卡斯尔大学）
Direct automatic segmentation of objects from 3D medical imaging, such as magnetic resonance (MR) imaging, is challenging as it often involves accurately identifying a number of individual objects with complex geometries within a large volume under investigation. To address these challenges, most deep learning approaches typically enhance their learning capability by substantially increasing the complexity or the number of trainable parameters within their models. Consequently, these models generally require long inference time on standard workstations operating clinical MR systems and are res…
Validation of Whole-Slide Foundation Models for Image Retrieval in TCGA Data
KIMIA Lab, Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN, USA（KIMIA实验室，人工智能与信息学，梅奥诊所，罗切斯特，明尼苏达州，美国）；Department of Neurology, Northwestern University Feinberg School of Medicine, Chicago, IL, USA（神经病学系，北western大学费因伯格医学院，芝加哥，伊利诺伊州，美国）；Department of Computer Science, Temple University, Philadelphia, PA, USA（计算机科学系，泰勒大学，费城，宾夕法尼亚州，美国）；Department of Breast and Melanoma Surgical Oncology, Comprehensive Cancer Center, Mayo Clinic, Rochester, MN, USA（乳腺和黑色素瘤外科肿瘤学系，综合癌症中心，梅奥诊所，罗切斯特，明尼苏达州，美国）；Department of Oncology, Comprehensive Cancer Center, Mayo Clinic, Rochester, MN, USA（肿瘤学系，综合癌症中心，梅奥诊所，罗切斯特，明尼苏达州，美国）；Department of Quantitative Health Sciences, Mayo Clinic, Rochester, MN, USA（定量健康科学系，梅奥诊所，罗切斯特，明尼苏达州，美国）
Bag of Image Patch Embedding Behind the Success of Self-Supervised Learning
Meta AI - FAIR（Meta人工智能基础研究实验室）；Center for Data Science, New York University（纽约大学数据科学中心）；Inria（法国国家信息与自动化研究所）；École normale supérieure（巴黎高等师范学院）；CNRS（法国国家科学研究中心）；PSL Research University（巴黎文理研究大学）；Redwood Center（红杉中心）；UC Berkeley（加州大学伯克利分校）；Courant Institute（柯朗研究所）
Traditional dictionary learning based CT reconstruction methods are patch-based and the features learned with these methods often contain shifted versions of the same features. To deal with these problems, the convolutional sparse coding (CSC) has been proposed and introduced into various applications. In this paper, inspired by the successful applications of CSC in the field of signal processing, we propose a novel sparse-view CT reconstruction method based on CSC with gradient regularization on feature maps. By directly working on whole image, which need not to divide the image into overlapp…
Hasso Plattner Institute for Digital Health at the Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院哈索·普拉特纳数字健康研究所）；Hasso Plattner Institute for Digital Engineering, University of Potsdam（波茨坦大学哈索·普拉特纳数字工程研究所）；Google Research（谷歌研究院）
High-resolution images are prevalent in various applications, such as autonomous driving and computer-aided diagnosis. However, training neural networks on such images is computationally challenging and easily leads to out-of-memory errors even on modern GPUs. We propose a simple method, Iterative Patch Selection (IPS), which decouples the memory usage from the input size and thus enables the processing of arbitrarily large images under tight hardware constraints. IPS achieves this by selecting only the most salient patches, which are then aggregated into a global representation for image reco…
The spread of social networking services has created an increasing demand for selecting, editing, and generating impressive images. This trend increases the importance of evaluating image aesthetics as a complementary function of automatic image processing. We propose a multi-patch method, named MPA-Net (Multi-Patch Aggregation Network), to predict image aesthetics scores by maintaining the original aspect ratios of contents in the images. Through an experiment involving the large-scale AVA dataset, which contains 250,000 images, we show that the effectiveness of the equal-interval multi-patch…
