Humans rely heavily on shapes as a primary cue for object recognition. As secondary cues, colours and textures are also beneficial in this regard. Convolutional neural networks (CNNs), an imitation of biological neural networks, have been shown to exhibit conflicting properties. Some studies indicate that CNNs are biased towards textures whereas, another set of studies suggests shape bias for a classification task. However, they do not discuss the role of colours, implying its possible humble role in the task of object recognition. In this paper, we empirically investigate the importance of colours in object recognition for CNNs. We are able to demonstrate that CNNs often rely heavily on colour information while making a prediction. Our results show that the degree of dependency on colours tend to vary from one dataset to another. Moreover, networks tend to rely more on colours if trained from scratch. Pre-training can allow the model to be less colour dependent. To facilitate these findings, we follow the framework often deployed in understanding role of colours in object recognition for humans. We evaluate a model trained with congruent images (images in original colours eg. red strawberries) on congruent, greyscale, and incongruent images (images in unnatural colours eg. blue strawberries). We measure and analyse network's predictive performance (top-1 accuracy) under these different stylisations. We utilise standard datasets of supervised image classification and fine-grained image classification in our experiments.
ORamaVR；University of Crete（克里特大学）；University of Western Macedonia（西马其顿大学）
We present a Convolutional Neural Network (CNN) suitable to assess unconstrained surgical cuttings, trained on a dataset created with a data augmentation technique.
Image colourisation is an ill-posed problem, with multiple correct solutions which depend on the context and object instances present in the input datum. Previous approaches attacked the problem either by requiring intense user interactions or by exploiting the ability of convolutional neural networks (CNNs) in learning image level (context) features. However, obtaining human hints is not always feasible and CNNs alone are not able to learn object-level semantics unless multiple models pretrained with supervision are considered. In this work, we propose a single network, named UCapsNet, that s…
Columbia University Irving Medical Center（哥伦比亚大学欧文医学中心）；Brown University（布朗大学）
We reproduced the results of CheXNet with fixed hyperparameters and 50 different random seeds to identify 14 finding in chest radiographs (x-rays). Because CheXNet fine-tunes a pre-trained DenseNet, the random seed affects the ordering of the batches of training data but not the initialized model weights. We found substantial variability in predictions for the same radiograph across model runs (mean ln[(maximum probability)/(minimum probability)] 2.45, coefficient of variation 0.543). This individual radiograph-level variability was not fully reflected in the variability of AUC on a large test…
It is known that humans display "shape bias" when classifying new items, i.e., they prefer to categorize objects based on their shape rather than color. Convolutional Neural Networks (CNNs) are also designed to take into account the spatial structure of image data. In fact, experiments on image datasets, consisting of triples of a probe image, a shape-match and a color-match, have shown that one-shot learning models display shape bias as well. In this paper, we examine the shape bias property of CNNs. In order to conduct large scale experiments, we propose using the model accuracy on images wi…
Assessing the Alignment of Popular CNNs to the Brain for Valence Appraisal
KU Leuven, De Nayer Campus, Dept. of Computer Science（KU莱文大学，Nayer校区，计算机科学系）；Leuven.AI - KU Leuven Institute for AI（Leuven.AI - KU莱文大学人工智能研究所）；Department of Brain and Cognition, Leuven Brain Institute, Faculty of Psychology & Educational Sciences（脑与认知系，Leuven脑研究所，心理学与教育科学学院）；Neuropsychiatry, Leuven Brain Institute KU Leuven（神经精神病学，Leuven脑研究所KU莱文大学）；Vrije Universiteit Brussel（布鲁塞尔自由大学）
Institute of Information Systems, HES-SO (University of Applied Sciences and Arts Western Switzerland)（瑞士西部应用科学与艺术大学 信息系统研究所）；Department of Computer Science, University of Geneva（日内瓦大学 计算机科学系）；Department of Neurosciences, University of Padua（帕多瓦大学 神经科学系）；Inselspital, Bern Hospital（伯尔尼大学医院）；Medical Faculty, University of Geneva（日内瓦大学 医学院）
Computational pathology is a domain that aims to develop algorithms to automatically analyze large digitized histopathology images, called whole slide images (WSI). WSIs are produced scanning thin tissue samples that are stained to make specific structures visible. They show stain colour heterogeneity due to different preparation and scanning settings applied across medical centers. Stain colour heterogeneity is a problem to train convolutional neural networks (CNN), the state-of-the-art algorithms for most computational pathology tasks, since CNNs usually underperform when tested on images in…
BBC Research & Development（英国广播公司研究与发展部）；Dublin City University（都柏林城市大学）
In this work recent advances in conditional adversarial networks are investigated to develop an end-to-end architecture based on Convolutional Neural Networks (CNNs) to directly map realistic colours to an input greyscale image. Observing that existing colourisation methods sometimes exhibit a lack of colourfulness, this paper proposes a method to improve colourisation results. In particular, the method uses Generative Adversarial Neural Networks (GANs) and focuses on improvement of training stability to enable better generalisation in large multi-class image datasets. Additionally, the integr…
University of Southampton（南安普顿大学）；Vision Learning and Control, Electronics and Computer Science, University of Southampton（南安普顿大学视觉学习与控制、电子与计算机科学（注：机构粒度要求到大学级别，此处完整机构为University of Southampton，其余为其内部部门，按要求提取到大学级别））
Recent work suggests that changing Convolutional Neural Network (CNN) architecture by introducing a bottleneck in the second layer can yield changes in learned function. To understand this relationship fully requires a way of quantitatively comparing trained networks. The fields of electrophysiology and psychophysics have developed a wealth of methods for characterising visual systems which permit such comparisons. Inspired by these methods, we propose an approach to obtaining spatial and colour tuning curves for convolutional neurons, which can be used to classify cells in terms of their spat…
Penn State University（宾夕法尼亚州立大学）；International Institute of Tropical Agriculture（国际热带农业研究所）
Convolutional neural network models (CNNs) have made major advances in computer vision tasks in the last five years. Given the challenge in collecting real world datasets, most studies report performance metrics based on available research datasets. In scenarios where CNNs are to be deployed on images or videos from mobile devices, models are presented with new challenges due to lighting, angle, and camera specifications, which are not accounted for in research datasets. It is essential for assessment to also be conducted on real world datasets if such models are to be reliably integrated with…
University of Copenhagen（哥本哈根大学）；Eindhoven University of Technology（埃因霍温理工大学）；Hvidovre Hospital（维兹奥勒医院）；Bispebjerg Hospital（比斯佩比约医院）；Erasmus MC - University Medical Center Rotterdam（伊拉斯谟医学中心——鹿特丹大学医学中心）
Supervised feature learning using convolutional neural networks (CNNs) can provide concise and disease relevant representations of medical images. However, training CNNs requires annotated image data. Annotating medical images can be a time-consuming task and even expert annotations are subject to substantial inter- and intra-rater variability. Assessing visual similarity of images instead of indicating specific pathologies or estimating disease severity could allow non-experts to participate, help uncover new patterns, and possibly reduce rater variability. We consider the task of assessing e…
Convolutional Neural Networks (CNNs) have become the state-of-the-art method to learn from image data. However, recent research shows that they may include a texture and colour bias in their representation, contrary to the intuition that they learn the shapes of the image content and to human biological learning. Thus, recent works have attempted to increase the shape bias in CNNs in order to train more robust and accurate networks on tasks. One such approach uses style-transfer in order to remove texture clues from the data. This work reproduces this methodology on four image classification d…
Understanding the 3D structure of a scene is of vital importance, when it comes to developing fully autonomous robots. To this end, we present a novel deep learning based framework that estimates depth, surface normals and surface curvature by only using a single RGB image. To the best of our knowledge this is the first work to estimate surface curvature from colour using a machine learning approach. Additionally, we demonstrate that by tuning the network to infer well designed features, such as surface curvature, we can achieve improved performance at estimating depth and normals.This indicat…
École de Technologie Supérieure, University of Québec（魁北克大学高等技术学院）；Pontifical Catholic University of Paraná（巴拉那天主教大学）
Facial expression recognition is a major problem in the domain of artificial intelligence. One of the best ways to solve this problem is the use of convolutional neural networks (CNNs). However, a large amount of data is required to train properly these networks but most of the datasets available for facial expression recognition are relatively small. A common way to circumvent the lack of data is to use CNNs trained on large datasets of different domains and fine-tuning the layers of such networks to the target domain. However, the fine-tuning process does not preserve the memory integrity as…
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；University College London（伦敦大学学院）
Image quality assessment (IQA) algorithm aims to quantify the human perception of image quality. Unfortunately, there is a performance drop when assessing the distortion images generated by generative adversarial network (GAN) with seemingly realistic texture. In this work, we conjecture that this maladaptation lies in the backbone of IQA models, where patch-level prediction methods use independent image patches as input to calculate their scores separately, but lack spatial relationship modeling among image patches. Therefore, we propose an Attention-based Hybrid Image Quality Assessment Netw…
Kitware, Inc.（Kitware公司）；University of Notre Dame（圣母大学）；SPRX
For the last decade, convolutional neural networks (CNNs) have vastly superseded their predecessors in nearly all vision tasks in artificial intelligence, including object recognition. However, despite abundant advancements, they continue to pale in comparison to biological vision. This chasm has prompted the development of biologically-inspired models that have attempted to mimic the human visual system, primarily at a neural level, which is evaluated using standard dataset benchmarks. However, more work is needed to understand how these models perceive the visual world. This article proposes…
University of Electronic Science and Technology of China（电子科技大学）
Many works have concentrated on visualizing and understanding the inner mechanism of convolutional neural networks (CNNs) by generating images that activate some specific neurons, which is called deep visualization. However, it is still unclear what the filters extract from images intuitively. In this paper, we propose a modified code inversion algorithm, called feature map inversion, to understand the function of filter of interest in CNNs. We reveal that every filter extracts a specific texture. The texture from higher layer contains more colours and more intricate structures. We also demons…
A Comparative Survey of Vision Transformers for Feature Extraction in Texture Analysis
São Carlos Institute of Physics, University of São Paulo（圣保罗大学圣卡洛斯物理学院）；Institute of Mathematics and Computer Sciences, University of São Paulo（圣保罗大学数学与计算机科学研究所）；Institute of Biosciences, Humanities and Exact Sciences, São Paulo State University（圣保罗州立大学生物科学、人文与精确科学研究所）；Ghent University（根特大学）
本文系统比较了21种预训练视觉Transformer在纹理特征提取上的表现，发现ViTs在鲁棒性、颜色/材料纹理和属性识别等任务中普遍优于CNNs和手工模型，并指出ViT-B with DINO、BeiTv2、Swin及EfficientFormer为有前景的模型。
Tensor network compressibility of convolutional models
Multiverse Computing（多元宇宙计算公司）；Donostia International Physics Center（多诺斯蒂亚国际物理中心）；Institute for Advanced Studies in Basic Sciences (IASBS)（基础科学高级研究所）；Ikerbasque Foundation for Science（伊克尔巴斯克科学基金会）
In this work we investigate how to achieve equivariance to input transformations in deep networks, purely from data, without being given a model of those transformations. Convolutional Neural Networks (CNNs), for example, are equivariant to image translation, a transformation that can be easily modelled (by shifting the pixels vertically or horizontally). Other transformations, such as out-of-plane rotations, do not admit a simple analytic model. We propose an auto-encoder architecture whose embedding obeys an arbitrary set of equivariance relations simultaneously, such as translation, rotatio…
In the recent past, algorithms based on Convolutional Neural Networks (CNNs) have achieved significant milestones in object recognition. With large examples of each object class, standard datasets train well for inter-class variability. However, gathering sufficient data to train for a particular instance of an object within a class is impractical. Furthermore, quantitatively assessing the imaging conditions for each image in a given dataset is not feasible. By generating sufficient images with known imaging conditions, we study to what extent CNNs can cope with hard imaging conditions for ins…
IIT Jodhpur（印度理工学院焦特布尔分校）；University of Bonn（波恩大学）
When judging style, a key question that often arises is whether or not a pair of objects are compatible with each other. In this paper we investigate how Siamese networks can be used efficiently for assessing the style compatibility between images of furniture items. We show that the middle layers of pretrained CNNs can capture essential information about furniture style, which allows for efficient applications of such networks for this task. We also use a joint image-text embedding method that allows for the querying of stylistically compatible furniture items, along with additional attribute…
Deeply Supervised Skin Lesions Diagnosis with Stage and Branch Attention
City University of Hong Kong（香港城市大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Shenzhen Research Institute, City University of Hong Kong（香港城市大学深圳研究院）
This paper presents an image classification based approach for skeleton-based video action recognition problem. Firstly, A dataset independent translation-scale invariant image mapping method is proposed, which transformes the skeleton videos to colour images, named skeleton-images. Secondly, A multi-scale deep convolutional neural network (CNN) architecture is proposed which could be built and fine-tuned on the powerful pre-trained CNNs, e.g., AlexNet, VGGNet, ResNet etal.. Even though the skeleton-images are very different from natural images, the fine-tune strategy still works well. At last…
University of Central Florida（中佛罗里达大学）；Purdue University（普渡大学）
Classification and clustering have been studied separately in machine learning and computer vision. Inspired by the recent success of deep learning models in solving various vision problems (e.g., object recognition, semantic segmentation) and the fact that humans serve as the gold standard in assessing clustering algorithms, here, we advocate for a unified treatment of the two problems and suggest that hierarchical frameworks that progressively build complex patterns on top of the simpler ones (e.g., convolutional neural networks) offer a promising solution. We do not dwell much on the learni…
Beijing Jiaotong University（北京交通大学）；University of South Carolina（南卡罗来纳大学）；Tianjin University（天津大学）
Shadow removal can significantly improve the image visual quality and has many applications in computer vision. Deep learning methods based on CNNs have become the most effective approach for shadow removal by training on either paired data, where both the shadow and underlying shadow-free versions of an image are known, or unpaired data, where shadow and shadow-free training images are totally different with no correspondence. In practice, CNN training on unpaired data is more preferred given the easiness of training data collection. In this paper, we present a new Lightness-Guided Shadow Rem…
The Academy for Mathematics, Science, and Engineering（数学、科学与工程学院）
Natural disasters ravage the world's cities, valleys, and shores on a regular basis. Deploying precise and efficient computational mechanisms for assessing infrastructure damage is essential to channel resources and minimize the loss of life. Using a dataset that includes labeled pre- and post- disaster satellite imagery, we take a machine learning-based remote sensing approach and train multiple convolutional neural networks (CNNs) to assess building damage on a per-building basis. We present a novel methodology of interpretable deep learning that seeks to explicitly investigate the most usef…
Human Body Dimensions Estimation (HBDE) is a task that an intelligent agent can perform to attempt to determine human body information from images (2D) or point clouds or meshes (3D). More specifically, if we define the HBDE problem as inferring human body measurements from images, then HBDE is a difficult, inverse, multi-task regression problem that can be tackled with machine learning techniques, particularly convolutional neural networks (CNN). Despite the community's tremendous effort to advance human shape analysis, there is a lack of systematic experiments to assess CNNs estimation of hu…
Medical University of Vienna（维也纳医科大学）；Loughborough University（拉夫堡大学）；TissueGnostics GmbH（TissueGnostics有限公司）
Foot ulcer is a common complication of diabetes mellitus and, associated with substantial morbidity and mortality, remains a major risk factor for lower leg amputations. Extracting accurate morphological features from foot wounds is crucial for appropriate treatment. Although visual inspection by a medical professional is the common approach for diagnosis, this is subjective and error-prone, and computer-aided approaches thus provide an interesting alternative. Deep learning-based methods, and in particular convolutional neural networks (CNNs), have shown excellent performance for various task…
