We introduce an approach for analyzing the variation of features generated by convolutional neural networks (CNNs) with respect to scene factors that occur in natural images. Such factors may include object style, 3D viewpoint, color, and scene lighting configuration. Our approach analyzes CNN feature responses corresponding to different scene factors by controlling for them via rendering using a large database of 3D CAD models. The rendered images are presented to a trained CNN and responses for different layers are studied with respect to the input scene factors. We perform a decomposition of the responses based on knowledge of the input scene factors and analyze the resulting components. In particular, we quantify their relative importance in the CNN responses and visualize them using principal component analysis. We show qualitative and quantitative results of our study on three CNNs trained on large image datasets: AlexNet, Places, and Oxford VGG. We observe important differences across the networks and CNN layers for different scene factors and object categories. Finally, we demonstrate that our analysis based on computer-generated imagery translates to the network representation of natural images.
The University of Melbourne（墨尔本大学）；Royal Netherlands Meteorological Institute (KNMI)（荷兰皇家气象研究所）；University of Melbourne（墨尔本大学）；Environment Protection Authority Victoria（维多利亚州环境保护局）
Analysis of overhead imagery using computer vision is a problem that has received considerable attention in academic literature. Most techniques that operate in this space are both highly specialised and require expensive manual annotation of large datasets. These problems are addressed here through the development of a more generic framework, incorporating advances in representation learning which allows for more flexibility in analysing new categories of imagery with limited labeled data. First, a robust representation of an unlabeled aerial imagery dataset was created based on the momentum …
Image forensics is an increasingly relevant problem, as it can potentially address online disinformation campaigns and mitigate problematic aspects of social media. Of particular interest, given its recent successes, is the detection of imagery produced by Generative Adversarial Networks (GANs), e.g. `deepfakes'. Leveraging large training sets and extensive computing resources, recent work has shown that GANs can be trained to generate synthetic imagery which is (in some ways) indistinguishable from real imagery. We analyze the structure of the generating network of a popular GAN implementatio…
Deep learning is revolutionizing the mapping industry. Under lightweight human curation, computer has generated almost half of the roads in Thailand on OpenStreetMap (OSM) using high-resolution aerial imagery. Bing maps are displaying 125 million computer-generated building polygons in the U.S. While tremendously more efficient than manual mapping, one cannot map out everything from the air. Especially for roads, a small prediction gap by image occlusion renders the entire road useless for routing. Misconnections can be more dangerous. Therefore computer-based mapping often requires local veri…
Studies evaluating bikeability usually compute spatial indicators shaping cycling conditions and conflate them in a quantitative index. Much research involves site visits or conventional geospatial approaches, and few studies have leveraged street view imagery (SVI) for conducting virtual audits. These have assessed a limited range of aspects, and not all have been automated using computer vision (CV). Furthermore, studies have not yet zeroed in on gauging the usability of these technologies thoroughly. We investigate, with experiments at a fine spatial scale and across multiple geographies (S…
North Dakota State University of Agriculture and Applied Science（北达科他州立大学农业与应用科学学院）
Currently, weed control in a corn field is performed by a blanket application of herbicides that do not consider spatial distribution information of weeds and also uses an extensive amount of chemical herbicides. To reduce the amount of chemicals, we used drone-based high-resolution imagery and computer-vision techniques to perform site-specific weed control in corn.
Accurate travel time estimation is paramount for providing transit users with reliable schedules and dependable real-time information. This paper is the first to utilize roadside urban imagery for direct transit travel time prediction. We propose and evaluate an end-to-end framework integrating traditional transit data sources with a roadside camera for automated roadside image data acquisition, labeling, and model training to predict transit travel times across a segment of interest. First, we show how the GTFS real-time data can be utilized as an efficient activation mechanism for a roadside…
University of Maryland Baltimore County（马里兰大学巴尔的摩县分校）；Texas A&M University（得克萨斯农工大学）
Visual scene understanding is the core task in making any crucial decision in any computer vision system. Although popular computer vision datasets like Cityscapes, MS-COCO, PASCAL provide good benchmarks for several tasks (e.g. image classification, segmentation, object detection), these datasets are hardly suitable for post disaster damage assessments. On the other hand, existing natural disaster datasets include mainly satellite imagery which have low spatial resolution and a high revisit period. Therefore, they do not have a scope to provide quick and efficient damage assessment tasks. Unm…
Major decisions from governments and other large organizations rely on measurements of the populace's well-being, but making such measurements at a broad scale is expensive and thus infrequent in much of the developing world. We propose an inexpensive, scalable, and interpretable approach to predict key livelihood indicators from public crowd-sourced street-level imagery. Such imagery can be cheaply collected and more frequently updated compared to traditional surveying methods, while containing plausibly relevant information for a range of livelihood indicators. We propose two approaches to l…
Machine learning is increasingly critical for analysis of the ever-growing corpora of overhead imagery. Advanced computer vision object detection techniques have demonstrated great success in identifying objects of interest such as ships, automobiles, and aircraft from satellite and drone imagery. Yet relying on computer vision opens up significant vulnerabilities, namely, the susceptibility of object detection algorithms to adversarial attacks. In this paper we explore the efficacy and drawbacks of adversarial camouflage in an overhead imagery context. While a number of recent papers have dem…
Universidad Publica de Navarra（纳瓦拉公立大学）；Institute of Smart Cities, Universidad Publica de Navarra（纳瓦拉公立大学智慧城市研究院）；Hospital Complex of Navarre (CHN)（纳瓦拉综合医院）；Slovak University of Technology in Bratislava（布拉迪斯拉发斯洛伐克工业大学）；University of Technology Sydney（悉尼科技大学）
In this work we study the use of moderate deviation functions to measure similarity and dissimilarity among a set of given interval-valued data. To do so, we introduce the notion of interval-valued moderate deviation function and we study in particular those interval-valued moderate deviation functions which preserve the width of the input intervals. Then, we study how to apply these functions to construct interval-valued aggregation functions. We have applied them in the decision making phase of two Motor-Imagery Brain Computer Interface frameworks, obtaining better results than those obtaine…
Brain-computer interface uses brain signals to communicate with external devices without actual control. Many studies have been conducted to classify motor imagery based on machine learning. However, classifying imagery data with sparse spatial characteristics, such as single-arm motor imagery, remains a challenge. In this paper, we proposed a method to factorize EEG signals into two groups to classify motor imagery even if spatial features are sparse. Based on adversarial learning, we focused on extracting common features of EEG signals which are robust to noise and extracting only signal fea…
Site-specific weed management in corn using UAS imagery analysis and computer vision techniques
Center for Precision & Automated Agricultural Systems（精准与自动化农业系统中心）；Washington State University（华盛顿州立大学）；Agricultural and Biosystems Engineering（农业与生物系统工程学院）；NDSU Carrington Research Extension Center（北达科他州立大学卡灵顿研究推广中心）；North Dakota State University（北达科他州立大学）
European Commission, Joint Research Centre (JRC)（欧盟委员会联合研究中心）
Street-level imagery holds a significant potential to scale-up in-situ data collection. This is enabled by combining the use of cheap high quality cameras with recent advances in deep learning compute solutions to derive relevant thematic information. We present a framework to collect and extract crop type and phenological information from street level imagery using computer vision. During the 2018 growing season, high definition pictures were captured with side-looking action cameras in the Flevoland province of the Netherlands. Each month from March to October, a fixed 200-km route was surve…
Urban planning applications (energy audits, investment, etc.) require an understanding of built infrastructure and its environment, i.e., both low-level, physical features (amount of vegetation, building area and geometry etc.), as well as higher-level concepts such as land use classes (which encode expert understanding of socio-economic end uses). This kind of data is expensive and labor-intensive to obtain, which limits its availability (particularly in developing countries). We analyze patterns in land use in urban neighborhoods using large-scale satellite imagery data (which is available w…
This paper investigates conditional generative adversarial networks (cGANs) to overcome a fundamental limitation of using geotagged media for geographic discovery, namely its sparse and uneven spatial distribution. We train a cGAN to generate ground-level views of a location given overhead imagery. We show the "fake" ground-level images are natural looking and are structurally similar to the real images. More significantly, we show the generated images are representative of the locations and that the representations learned by the cGANs are informative. In particular, we show that dense featur…
Raisa Energy LLC；Faculty of Engineering, Cairo University（开罗大学工程学院）；Faculty of Computers and Information, Cairo University（开罗大学计算机与信息学院）
Automation of objects labeling in aerial imagery is a computer vision task with numerous practical applications. Fields like energy exploration require an automated method to process a continuous stream of imagery on a daily basis. In this paper we propose a pipeline to tackle this problem using a stack of convolutional neural networks (U-Net architecture) arranged end-to-end. Each network works as post-processor to the previous one. Our model outperforms current state-of-the-art on two different datasets: Inria Aerial Image Labeling dataset and Massachusetts Buildings dataset each with differ…
Chester F. Carlson Center for Imaging Science Rochester Institute of Technology（切斯特·F·卡尔森成像科学中心 罗切斯特理工学院）
Deep convolutional neural networks (DCNNs) have been used to achieve state-of-the-art performance on many computer vision tasks (e.g., object recognition, object detection, semantic segmentation) thanks to a large repository of annotated image data. Large labeled datasets for other sensor modalities, e.g., multispectral imagery (MSI), are not available due to the large cost and manpower required. In this paper, we adapt state-of-the-art DCNN frameworks in computer vision for semantic segmentation for MSI imagery. To overcome label scarcity for MSI data, we substitute real MSI for generated syn…
Carnegie Mellon University（卡内基梅隆大学）；Software Engineering Institute（软件工程研究所）；Defense Innovation Unit（国防创新部门）；Department of Defense（国防部）；CrowdAI, Inc.（CrowdAI公司）
We present xBD, a new, large-scale dataset for the advancement of change detection and building damage assessment for humanitarian assistance and disaster recovery research. Natural disaster response requires an accurate understanding of damaged buildings in an affected region. Current response strategies require in-person damage assessments within 24-48 hours of a disaster. Massive potential exists for using aerial imagery combined with computer vision algorithms to assess damage and reduce the potential danger to human life. In collaboration with multiple disaster response agencies, xBD prov…
Brain-computer interface (BCI) is used for communication between humans and devices by recognizing status and intention of humans. Communication between humans and a drone using electroencephalogram (EEG) signals is one of the most challenging issues in the BCI domain. In particular, the control of drone swarms (the direction and formation) has more advantages compared to the control of a drone. The visual imagery (VI) paradigm is that subjects visually imagine specific objects or scenes. Reduction of the variability among EEG signals of subjects is essential for practical BCI-based systems. I…
Detection of small objects in large swaths of imagery is one of the primary problems in satellite imagery analytics. While object detection in ground-based imagery has benefited from research into new deep learning approaches, transitioning such technology to overhead imagery is nontrivial. Among the challenges is the sheer number of pixels and geographic extent per image: a single DigitalGlobe satellite image encompasses >64 km2 and over 250 million pixels. Another challenge is that objects of interest are minuscule (often only ~10 pixels in extent), which complicates traditional computer vis…
Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）；Descartes Labs, Inc（笛卡尔实验室公司）
In this work we demonstrate that generative adversarial networks (GANs) can be used to generate realistic pervasive changes in remote sensing imagery, even in an unpaired training setting. We investigate some transformation quality metrics based on deep embedding of the generated and real images which enable visualization and understanding of the training dynamics of the GAN, and may provide a useful measure in terms of quantifying how distinguishable the generated images are from real images. We also identify some artifacts introduced by the GAN in the generated images, which are likely to co…
Technische Universität Berlin（柏林工业大学）；DFKI Projektbüro Berlin（德国人工智能研究中心柏林项目办公室）
Video gaming streaming services are growing rapidly due to new services such as passive video streaming, e.g. Twitch.tv, and cloud gaming, e.g. Nvidia Geforce Now. In contrast to traditional video content, gaming content has special characteristics such as extremely high motion for some games, special motion patterns, synthetic content and repetitive content, which makes the state-of-the-art video and image quality metrics perform weaker for this special computer generated content. In this paper, we outline our plan to build a deep learningbased quality metric for video gaming quality assessme…
Deep hybrid model with satellite imagery: how to combine demand modeling and computer vision for behavior analysis?
Massachusetts Institute of Technology（麻省理工学院）；University of California at Berkeley（加州大学伯克利分校）；The University of Hong Kong（香港大学）；University of Florida（佛罗里达大学）
Space and Naval Warfare Systems Center Pacific（太平洋空间与海战系统中心）
Maritime collisions involving multiple ships are considered rare, but in 2017 several United States Navy vessels were involved in fatal at-sea collisions that resulted in the death of seventeen American Servicemembers. The experimentation introduced in this paper is a direct response to these incidents. We propose a shipboard Collision-At-Sea avoidance system, based on video image processing, that will help ensure the safe stationing and navigation of maritime vessels. Our system leverages a convolutional neural network trained on synthetic maritime imagery in order to detect nearby vessels wi…
The usage of Unmanned Aerial Vehicles (UAVs) in the context of structural health inspection is recently gaining tremendous popularity. Camera mounted UAVs enable the fast acquisition of a large number of images often used for mapping, 3D model reconstruction, and as an assisting tool for inspectors. Due to the number of images captured during large scale UAV surveys, a manual image-based inspection analysis of entire assets cannot be efficiently performed by qualified engineers. Additionally, comparing defects to past inspections requires the retrieval of relevant images which is often impract…
This work evaluates the reproducibility of the paper "CNN-generated images are surprisingly easy to spot... for now" by Wang et al. published at CVPR 2020. The paper addresses the challenge of detecting CNN-generated imagery, which has reached the potential to even fool humans. The authors propose two methods which help an image classifier to generalize from being trained on one specific CNN to detecting imagery produced by unseen architectures, training methods, or data sets. The paper proposes two methods to help a classifier generalize: (i) utilizing different kinds of data augmentations an…
In this paper we propose a method for improving pedestrian detection in the thermal domain using two stages: first, a generative data augmentation approach is used, then a domain adaptation method using generated data adapts an RGB pedestrian detector. Our model, based on the Least-Squares Generative Adversarial Network, is trained to synthesize realistic thermal versions of input RGB images which are then used to augment the limited amount of labeled thermal pedestrian images available for training. We apply our generative data augmentation strategy in order to adapt a pretrained YOLOv3 pedes…
Israel Ministry Of Defense’s Directorate of Defense Research and Development (IMOD’s DDR&D)（以色列国防部国防研究与开发局 (IMOD DDR&D)）
Detection and classification of objects in overhead images are two important and challenging problems in computer vision. Among various research areas in this domain, the task of fine-grained classification of objects in overhead images has become ubiquitous in diverse real-world applications, due to recent advances in high-resolution satellite and airborne imaging systems. The small inter-class variations and the large intra class variations caused by the fine grained nature make it a challenging task, especially in low-resource cases. In this paper, we introduce COFGA a new open dataset for …
Massachusetts Institute of Technology（麻省理工学院）；University of California, Berkeley（加州大学伯克利分校）
The classification of streets on road networks has been focused on the vehicular transportational features of streets such as arterials, major roads, minor roads and so forth based on their transportational use. City authorities on the other hand have been shifting to more urban inclusive planning of streets, encompassing the side use of a street combined with the transportational features of a street. In such classification schemes, streets are labeled for example as commercial throughway, residential neighborhood, park etc. This modern approach to urban planning has been adopted by major cit…
Plant breeding programs extensively monitor the evolution of seed kernels for seed certification, wherein lies the need to appropriately label the seed kernels by type and quality. However, the breeding environments are large where the monitoring of seed kernels can be challenging due to the minuscule size of seed kernels. The use of unmanned aerial vehicles aids in seed monitoring and labeling since they can capture images at low altitudes whilst being able to access even the remotest areas in the environment. A key bottleneck in the labeling of seeds using UAV imagery is drone altitude i.e. …
Vehicle detection from GSV imagery: Predicting travel behaviour for cycling and motorcycling using Computer Vision
University of Cambridge（剑桥大学）；Indian Institute of Technology Delhi（德里印度理工学院）；University of Melbourne（墨尔本大学）；Utrecht University（乌得勒支大学）；University of Bristol（布里斯托尔大学）
本研究利用深度学习对全球185个城市的Google Street View图像进行车辆检测，结合beta回归模型预测骑行与摩托车出行方式份额，R²分别达0.614和0.612，为缺乏调查数据的城市提供了出行行为估算。
Accurate and fine-grained information about the extent of damage to buildings is essential for directing Humanitarian Aid and Disaster Response (HADR) operations in the immediate aftermath of any natural calamity. In recent years, satellite and UAV (drone) imagery has been used for this purpose, sometimes aided by computer vision algorithms. Existing Computer Vision approaches for building damage assessment typically rely on a two stage approach, consisting of building detection using an object detection model, followed by damage assessment through classification of the detected building tiles…
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；FZI Research Center for Information Technology（FZI信息技术研究中心）
In this work, we present LocGAN, our localization approach based on a geo-referenced aerial imagery and LiDAR grid maps. Currently, most self-localization approaches relate the current sensor observations to a map generated from previously acquired data. Unfortunately, this data is not always available and the generated maps are usually sensor setup specific. Global Navigation Satellite Systems (GNSS) can overcome this problem. However, they are not always reliable especially in urban areas due to multi-path and shadowing effects. Since aerial imagery is usually available, we can use it as pri…
Estimating the Impact of COVID-19 on Travel Demand in Houston Area Using Deep Learning and Satellite Imagery
Civil and Environmental Engineering, University of Houston（休斯顿大学土木与环境工程系）；Department of Transportation, Logistics and Finance, North Dakota State University（北达科他州立大学交通、物流与金融系）；School of Traffic and Transportation, Shijiazhuang Tiedao University（石家庄铁道大学交通运输学院）
University of British Columbia（英属哥伦比亚大学）；Department of Statistics, University of British Columbia（英属哥伦比亚大学统计系）；Department of Electrical and Computer Engineering, University of British Columbia（英属哥伦比亚大学电气与计算机工程系）；School of Information and Communications Engineering, Xi’an Jiaotong University（西安交通大学信息与通信工程学院）
Recently, generative adversarial networks (GANs) can generate photo-realistic fake facial images which are perceptually indistinguishable from real face photos, promoting research on fake face detection. Though fake face forensics can achieve high detection accuracy, their anti-forensic counterparts are less investigated. Here we explore more \textit{imperceptible} and \textit{transferable} anti-forensics for fake face imagery detection based on adversarial attacks. Since facial and background regions are often smooth, even small perturbation could cause noticeable perceptual impairment in fak…
Recent advances in computer vision using deep learning with RGB imagery (e.g., object recognition and detection) have been made possible thanks to the development of large annotated RGB image datasets. In contrast, multispectral image (MSI) and hyperspectral image (HSI) datasets contain far fewer labeled images, in part due to the wide variety of sensors used. These annotations are especially limited for semantic segmentation, or pixel-wise classification, of remote sensing imagery because it is labor intensive to generate image annotations. Low-shot learning algorithms can make effective infe…
The parsing of windows in building facades is a long-desired but challenging task in computer vision. It is crucial to urban analysis, semantic reconstruction, lifecycle analysis, digital twins, and scene parsing amongst other building-related tasks that require high-quality semantic data. This article investigates the usage of the mask R-CNN framework to be used for window detection of facade imagery input. We utilize transfer learning to train our proposed method on COCO weights with our own collected dataset of street view images of facades to produce instance segmentations of our new windo…
Stanford University（斯坦福大学）；UC Berkeley（加州大学伯克利分校）；The California Institute of Technology（加州理工学院）
This paper introduces a new, highly consequential setting for the use of computer vision for environmental sustainability. Concentrated Animal Feeding Operations (CAFOs) (aka intensive livestock farms or "factory farms") produce significant manure and pollution. Dumping manure in the winter months poses significant environmental risks and violates environmental law in many states. Yet the federal Environmental Protection Agency (EPA) and state agencies have relied primarily on self-reporting to monitor such instances of "land application." Our paper makes four contributions. First, we introduc…
Mission San Jose High School（米慎圣何塞高中）；Stanford Univeristy（斯坦福大学）
Determining the poverty levels of various regions throughout the world is crucial in identifying interventions for poverty reduction initiatives and directing resources fairly. However, reliable data on global economic livelihoods is hard to come by, especially for areas in the developing world, hampering efforts to both deploy services and monitor/evaluate progress. This is largely due to the fact that this data is obtained from traditional door-to-door surveys, which are time consuming and expensive. Overhead satellite imagery contain characteristics that make it possible to estimate the reg…
Gaining timely and reliable situation awareness after hazard events such as a hurricane is crucial to emergency managers and first responders. One effective way to achieve that goal is through damage assessment. Recently, disaster researchers have been utilizing imagery captured through satellites or drones to quantify the number of flooded/damaged buildings. In this paper, we propose a mixed data approach, which leverages publicly available satellite imagery and geolocation features of the affected area to identify damaged buildings after a hurricane. The method demonstrated significant impro…
University of South Carolina（南卡罗来纳大学）；East China Normal University（华东师范大学）
Nighttime satellite imagery has been applied in a wide range of fields. However, our limited understanding of how observed light intensity is formed and whether it can be simulated greatly hinders its further application. This study explores the potential of conditional Generative Adversarial Networks (cGAN) in translating multispectral imagery to nighttime imagery. A popular cGAN framework, pix2pix, was adopted and modified to facilitate this translation using gridded training image pairs derived from Landsat 8 and Visible Infrared Imaging Radiometer Suite (VIIRS). The results of this study p…
Global Fishing Watch（全球渔业观察）；Cambrio；Defense Innovation Unit（国防创新单元）；UC Berkeley（加州大学伯克利分校）
Unsustainable fishing practices worldwide pose a major threat to marine resources and ecosystems. Identifying vessels that do not show up in conventional monitoring systems -- known as ``dark vessels'' -- is key to managing and securing the health of marine environments. With the rise of satellite-based synthetic aperture radar (SAR) imaging and modern machine learning (ML), it is now possible to automate detection of dark vessels day or night, under all-weather conditions. SAR images, however, require a domain-specific treatment and are not widely accessible to the ML community. Maritime obje…
