In this paper we propose an adversarial generative grammar model for future prediction. The objective is to learn a model that explicitly captures temporal dependencies, providing a capability to forecast multiple, distinct future activities. Our adversarial grammar is designed so that it can learn stochastic production rules from the data distribution, jointly with its latent non-terminal representations. Being able to select multiple production rules during inference leads to different predicted outcomes, thus efficiently modeling many plausible futures. The adversarial generative grammar is evaluated on the Charades, MultiTHUMOS, Human3.6M, and 50 Salads datasets and on two activity prediction tasks: future 3D human pose prediction and future activity prediction. The proposed adversarial grammar outperforms the state-of-the-art approaches, being able to predict much more accurately and further in the future, than prior work.
University of California, Los Angeles（加州大学洛杉矶分校）；Xi’an Jiaotong University（西安交通大学）
This paper presents a novel method to predict future human activities from partially observed RGB-D videos. Human activity prediction is generally difficult due to its non-Markovian property and the rich context between human and environments. We use a stochastic grammar model to capture the compositional structure of events, integrating human actions, objects, and their affordances. We represent the event by a spatial-temporal And-Or graph (ST-AOG). The ST-AOG is composed of a temporal stochastic grammar defined on sub-activities, and spatial graphs representing sub-activities that consist of…
Temporally localizing activities within untrimmed videos has been extensively studied in recent years. Despite recent advances, existing methods for weakly-supervised temporal activity localization struggle to recognize when an activity is not occurring. To address this issue, we propose a novel method named A2CL-PT. Two triplets of the feature space are considered in our approach: one triplet is used to learn discriminative features for each activity class, and the other one is used to distinguish the features where no activity occurs (i.e. background features) from activity-related features …
This article proposes a novel approach for augmenting generative adversarial network (GAN) with a self-supervised task in order to improve its ability for encoding video representations that are useful in downstream tasks such as human activity recognition. In the proposed method, input video frames are randomly transformed by different spatial transformations, such as rotation, translation and shearing or temporal transformations such as shuffling temporal order of frames. Then discriminator is encouraged to predict the applied transformation by introducing an auxiliary loss. Subsequently, re…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Peking University（北京大学）
Future predictions on sequence data (e.g., videos or audios) require the algorithms to capture non-Markovian and compositional properties of high-level semantics. Context-free grammars are natural choices to capture such properties, but traditional grammar parsers (e.g., Earley parser) only take symbolic sentences as inputs. In this paper, we generalize the Earley parser to parse sequence data which is neither segmented nor labeled. This generalized Earley parser integrates a grammar parser with a classifier to find the optimal segmentation and labels, and makes top-down future predictions. Ex…
Reconstructing visual stimulus (image) only from human brain activity measured with functional Magnetic Resonance Imaging (fMRI) is a significant and meaningful task in Human-AI collaboration. However, the inconsistent distribution and representation between fMRI signals and visual images cause the heterogeneity gap. Moreover, the fMRI data is often extremely high-dimensional and contains a lot of visually-irrelevant information. Existing methods generally suffer from these issues so that a satisfactory reconstruction is still challenging. In this paper, we show that it is possible to overcome…
Deep Adversarial Learning with Activity-Based User Discrimination Task for Human Activity Recognition
AASS, Örebro University（厄勒布鲁大学AASS研究中心）；Faculty of Information Science and Electrical Engineering, Kyushu University（九州大学情报科学与电气工程学院）；ETSIDI, Universidad Politéctina de Madrid（马德里理工大学高等航空工程技术学院）
本文提出含新型基于活动的用户判别任务的对抗深度学习框架，以缓解 HAR 中的人际差异，并在三个数据集的 LOOCV 基准上取得优于以往方法和任务的分类效果。
Understanding the Vulnerability of Skeleton-based Human Activity Recognition via Black-box Attack
School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；University College London（伦敦大学学院）；Zhejiang University（浙江大学）；University of Bath（巴斯大学）；School of Computing, University of Leeds（利兹大学计算学院）
Georgia Institute of Technology（佐治亚理工学院）；Google（谷歌）
We propose an action recognition framework using Gen- erative Adversarial Networks. Our model involves train- ing a deep convolutional generative adversarial network (DCGAN) using a large video activity dataset without la- bel information. Then we use the trained discriminator from the GAN model as an unsupervised pre-training step and fine-tune the trained discriminator model on a labeled dataset to recognize human activities. We determine good network architectural and hyperparameter settings for us- ing the discriminator from DCGAN as a trained model to learn useful representations for acti…
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Technische Universität Kaiserslautern（凯泽斯劳滕工业大学）
Wearable sensor-based human activity recognition (HAR) has emerged as a principal research area and is utilized in a variety of applications. Recently, deep learning-based methods have achieved significant improvement in the HAR field with the development of human-computer interaction applications. However, they are limited to operating in a local neighborhood in the process of a standard convolution neural network, and correlations between different sensors on body positions are ignored. In addition, they still face significant challenging problems with performance degradation due to large ga…
Black-box Attacks on Image Activity Prediction and its Natural Language Explanations
Idiap Research Institute（伊迪亚普研究所）；University of Perugia（佩鲁贾大学）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；Queen Mary University of London（伦敦玛丽女王大学）
Northwestern Polytechnical University（西北工业大学）；University of Georgia（佐治亚大学）；The University of Texas at Arlington（德克萨斯大学阿灵顿分校）
Visual attention is a fundamental mechanism in the human brain, and it inspires the design of attention mechanisms in deep neural networks. However, most of the visual attention studies adopted eye-tracking data rather than the direct measurement of brain activity to characterize human visual attention. In addition, the adversarial relationship between the attention-related objects and attention-neglected background in the human visual system was not fully exploited. To bridge these gaps, we propose a novel brain-inspired adversarial visual attention network (BI-AVAN) to characterize human vis…
Data augmentation is a widely used technique in classification to increase data used in training. It improves generalization and reduces amount of annotated human activity data needed for training which reduces labour and time needed with the dataset. Sensor time-series data, unlike images, cannot be augmented by computationally simple transformation algorithms. State of the art models like Recurrent Generative Adversarial Networks (RGAN) are used to generate realistic synthetic data. In this paper, transformer based generative adversarial networks which have global attention on data, are comp…
The performance of Human Activity Recognition (HAR) models, particularly deep neural networks, is highly contingent upon the availability of the massive amount of annotated training data which should be sufficiently labeled. Though, data acquisition and manual annotation in the HAR domain are prohibitively expensive due to skilled human resource requirements in both steps. Hence, domain adaptation techniques have been proposed to adapt the knowledge from the existing source of data. More recently, adversarial transfer learning methods have shown very promising results in image classification, …
Given their substantial success in addressing a wide range of computer vision challenges, Convolutional Neural Networks (CNNs) are increasingly being used in smart home applications, with many of these applications relying on the automatic recognition of human activities. In this context, low-power radar devices have recently gained in popularity as recording sensors, given that the usage of these devices allows mitigating a number of privacy concerns, a key issue when making use of conventional video cameras. Another concern that is often cited when designing smart home applications is the re…
Dual-Attention and Adversarial Transfer Networks for Sim-to-Real Cross-Orientation Wireless Sensing
National Graduate College for Engineers, Southern University of Science and Technology（南方科技大学国家工程师学院）；Harbin Institute of Technology（哈尔滨工业大学）；Southern University of Science and Technology（南方科技大学）
Okayama University（冈山大学）；Japan Science and Technology Agency（日本科学技术振兴机构）；Rikkyo University（立教大学）；National Institute for Physiological Sciences（生理学研究所）；Araya Inc.（Araya 公司）；Keio University（庆应义塾大学）
Deep neural networks (DNNs) can accurately decode task-related information from brain activations. However, because of the nonlinearity of the DNN, the decisions made by DNNs are hardly interpretable. One of the promising approaches for explaining such a black-box system is counterfactual explanation. In this framework, the behavior of a black-box system is explained by comparing real data and realistic synthetic data that are specifically generated such that the black-box system outputs an unreal outcome. Here we introduce a novel generative DNN (counterfactual activation generator, CAG) that…
University of Leeds（利兹大学）；Southwest Jiaotong University（西南交通大学）；Zhejiang University（浙江大学）；University of Bath（巴斯大学）
Skeletal motion plays a vital role in human activity recognition as either an independent data source or a complement. The robustness of skeleton-based activity recognizers has been questioned recently, which shows that they are vulnerable to adversarial attacks when the full-knowledge of the recognizer is accessible to the attacker. However, this white-box requirement is overly restrictive in most scenarios and the attack is not truly threatening. In this paper, we show that such threats do exist under black-box settings too. To this end, we propose the first black-box adversarial attack meth…
Epileptic seizures are caused by abnormal, overly syn- chronized, electrical activity in the brain. The abnor- mal electrical activity manifests as waves, propagating across the brain. Accurate prediction of the propagation velocity and direction of these waves could enable real- time responsive brain stimulation to suppress or prevent the seizures entirely. However, this problem is very chal- lenging because the algorithm must be able to predict the neural signals in a sufficiently long time horizon to allow enough time for medical intervention. We consider how to accomplish long term predict…
University of Leeds（利兹大学）；Hefei University of Technology（合肥工业大学）；Institute of Deep Learning, Baidu Research（百度研究院深度学习研究所）
Skeletal motions have been heavily replied upon for human activity recognition (HAR). Recently, a universal vulnerability of skeleton-based HAR has been identified across a variety of classifiers and data, calling for mitigation. To this end, we propose the first black-box defense method for skeleton-based HAR to our best knowledge. Our method is featured by full Bayesian treatments of the clean data, the adversaries and the classifier, leading to (1) a new Bayesian Energy-based formulation of robust discriminative classifiers, (2) a new adversary sampling scheme based on natural motion manifo…
National Digital Switching System Engineering and Technological Research Center（国家数字交换系统工程技术研究中心）；People's Hospital of Henan Province（河南省人民医院）；Shanghai Mental Health Center, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属精神卫生中心）；Institute of Psychology and Behavioral Science, Shanghai Jiao Tong University（上海交通大学心理与行为科学研究院）
Despite the remarkable similarities between convolutional neural networks (CNN) and the human brain, CNNs still fall behind humans in many visual tasks, indicating that there still exist considerable differences between the two systems. Here, we leverage adversarial noise (AN) and adversarial interference (AI) images to quantify the consistency between neural representations and perceptual outcomes in the two systems. Humans can successfully recognize AI images as corresponding categories but perceive AN images as meaningless noise. In contrast, CNNs can correctly recognize AN images but mista…
Reconstructing perceived images from human brain activity monitored by functional magnetic resonance imaging (fMRI) is hard, especially for natural images. Existing methods often result in blurry and unintelligible reconstructions with low fidelity. In this study, we present a novel approach for enhanced image reconstruction, in which existing methods for object decoding and image reconstruction are merged together. This is achieved by conditioning the reconstructed image to its decoded image category using a class-conditional generative adversarial network and neural style transfer. The resul…
SOS: A Shuffle Order Strategy for Data Augmentation in Industrial Human Activity Recognition
Faculty of Computer Science（计算机科学系）；Engineering Ho Chi Minh city University of Technology- Vietnam National University Ho Chi Minh City, Vietnam（工程学院-胡志明市技术大学-越南国家大学胡志明市）；Department of Biomedical Engineering Ho Chi Minh city University of Technology- Vietnam National University Ho Chi Minh City, Vietnam（生物医学工程系-胡志明市技术大学-越南国家大学胡志明市）
UltraDfeGAN: Detail-Enhancing Generative Adversarial Networks for High-Fidelity Functional Ultrasound Synthesis
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Department of Information and Communication, Institute of Technology of Cambodia（柬埔寨技术学院信息与通信系）；Department of Telecommunication and Network Engineering, Institute of Technology of Cambodia（柬埔寨技术学院电信与网络工程系）
We address the problem of generating 3D human motions in dyadic activities. In contrast to the concurrent works, which mainly focus on generating the motion of a single actor from the textual description, we generate the motion of one of the actors from the motion of the other participating actor in the action. This is a particularly challenging, under-explored problem, that requires learning intricate relationships between the motion of two actors participating in an action and also identifying the action from the motion of one actor. To address these, we propose partner conditioned motion op…
PLA strategy support force information engineering university（中国人民解放军战略支援部队信息工程大学）
In the visual decoding domain, visually reconstructing presented images given the corresponding human brain activity monitored by functional magnetic resonance imaging (fMRI) is difficult, especially when reconstructing viewed natural images. Visual reconstruction is a conditional image generation on fMRI data and thus generative adversarial network (GAN) for natural image generation is recently introduced for this task. Although GAN-based methods have greatly improved, the fidelity and naturalness of reconstruction are still unsatisfactory due to the small number of fMRI data samples and the …
