Behavior Recognition Based on the Integration of Multigranular Motion Features
University of Electronic Science and Technology of China（电子科技大学）；Beijing Aerospace Institute for Metrology and Measurement Technology（北京航天计量测试技术研究所）；Southwestern University of Finance and Economics（西南财经大学）
The recognition of behaviors in videos usually requires a combinatorial analysis of the spatial information about objects and their dynamic action information in the temporal dimension. Specifically, behavior recognition may even rely more on the modeling of temporal information containing short-range and long-range motions; this contrasts with computer vision tasks involving images that focus on the understanding of spatial information. However, current solutions fail to jointly and comprehensively analyze short-range motion between adjacent frames and long-range temporal aggregations at large scales in videos. In this paper, we propose a novel behavior recognition method based on the integration of multigranular (IMG) motion features. In particular, we achieve reliable motion information modeling through the synergy of a channel attention-based short-term motion feature enhancement module (CMEM) and a cascaded long-term motion feature integration module (CLIM). We evaluate our model on several action recognition benchmarks such as HMDB51, Something-Something and UCF101. The experimental results demonstrate that our approach outperforms the previous state-of-the-art methods, which confirms its effectiveness and efficiency.
Facebook（脸书）；Zhejiang University（浙江大学）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）；NVIDIA（英伟达）
Most of the top performing action recognition methods use optical flow as a "black box" input. Here we take a deeper look at the combination of flow and action recognition, and investigate why optical flow is helpful, what makes a flow method good for action recognition, and how we can make it better. In particular, we investigate the impact of different flow algorithms and input transformations to better understand how these affect a state-of-the-art action recognition method. Furthermore, we fine tune two neural-network flow methods end-to-end on the most widely used action recognition datas…
Malong Technologies（马泷科技）；Shenzhen Malong Artificial Intelligence Research Center（深圳马泷人工智能研究中心）；State Key Lab for Novel Software Technology, Nanjing University（南京大学计算机科学与技术系（新型软件技术国家重点实验室））
In this work, we propose Knowledge Integration Networks (referred as KINet) for video action recognition. KINet is capable of aggregating meaningful context features which are of great importance to identifying an action, such as human information and scene context. We design a three-branch architecture consisting of a main branch for action recognition, and two auxiliary branches for human parsing and scene recognition which allow the model to encode the knowledge of human and scene for action recognition. We explore two pre-trained models as teacher networks to distill the knowledge of human…
Most current action recognition methods heavily rely on appearance information by taking an RGB sequence of entire image regions as input. While being effective in exploiting contextual information around humans, e.g., human appearance and scene category, they are easily fooled by out-of-context action videos where the contexts do not exactly match with target actions. In contrast, pose-based methods, which take a sequence of human skeletons only as input, suffer from inaccurate pose estimation or ambiguity of human pose per se. Integrating these two approaches has turned out to be non-trivial…
SenseTime Research（商汤科技研究院）；Chongqing University（重庆大学）；Shanghai AI Laboratory（上海人工智能实验室）
Efficient spatiotemporal modeling is an important yet challenging problem for video action recognition. Existing state-of-the-art methods exploit neighboring feature differences to obtain motion clues for short-term temporal modeling with a simple convolution. However, only one local convolution is incapable of handling various kinds of actions because of the limited receptive field. Besides, action-irrelated noises brought by camera movement will also harm the quality of extracted motion features. In this paper, we propose a Temporal Saliency Integration (TSI) block, which mainly contains a S…
Shanghai Jiao Tong University（上海交通大学）；Nanjing University（南京大学）；University of Chinese Academy of Sciences（中国科学院大学）
Action recognition is an important yet challenging task in computer vision. In this paper, we propose a novel deep-based framework for action recognition, which improves the recognition accuracy by: 1) deriving more precise features for representing actions, and 2) reducing the asynchrony between different information streams. We first introduce a coarse-to-fine network which extracts shared deep features at different action class granularities and progressively integrates them to obtain a more accurate feature representation for input actions. We further introduce an asynchronous fusion netwo…
University of Queensland（昆士兰大学）；Queensland University of Technology（昆士兰科技大学）；NICTA（澳大利亚国家信息通信技术中心）；Data61, CSIRO（澳大利亚联邦科学与工业研究组织 Data61）
We propose a hierarchical approach to multi-action recognition that performs joint classification and segmentation. A given video (containing several consecutive actions) is processed via a sequence of overlapping temporal windows. Each frame in a temporal window is represented through selective low-level spatio-temporal features which efficiently capture relevant local dynamics. Features from each window are represented as a Fisher vector, which captures first and second order statistics. Instead of directly classifying each Fisher vector, it is converted into a vector of class probabilities.…
University of Science and Technology of China（中国科学技术大学）；HERE Technologies（HERE科技公司）
Visual attributes in individual video frames, such as the presence of characteristic objects and scenes, offer substantial information for action recognition in videos. With individual 2D video frame as input, visual attributes extraction could be achieved effectively and efficiently with more sophisticated convolutional neural network than current 3D CNNs with spatio-temporal filters, thanks to fewer parameters in 2D CNNs. In this paper, the integration of visual attributes (including detection, encoding and classification) into multi-stream 3D CNN is proposed for action recognition in trimme…
Stanford University（斯坦福大学）；Massachusetts Institute of Technology（麻省理工学院）；Purdue University（普渡大学）
We present a system that demonstrates how the compositional structure of events, in concert with the compositional structure of language, can interplay with the underlying focusing mechanisms in video action recognition, thereby providing a medium, not only for top-down and bottom-up integration, but also for multi-modal integration between vision and language. We show how the roles played by participants (nouns), their characteristics (adjectives), the actions performed (verbs), the manner of such actions (adverbs), and changing spatial relations between participants (prepositions) in the for…
Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）
Fine-grained human action recognition is a core research topic in computer vision. Inspired by the recently proposed hierarchy representation of fine-grained actions in FineGym and SlowFast network for action recognition, we propose a novel multi-task network which exploits the FineGym hierarchy representation to achieve effective joint learning and prediction for fine-grained human action recognition. The multi-task network consists of three pathways of SlowOnly networks with gradually increased frame rates for events, sets and elements of fine-grained actions, followed by our proposed integr…
Human action Recognition for unknown views is a challenging task. We propose a view-invariant deep human action recognition framework, which is a novel integration of two important action cues: motion and shape temporal dynamics (STD). The motion stream encapsulates the motion content of action as RGB Dynamic Images (RGB-DIs) which are processed by the fine-tuned InceptionV3 model. The STD stream learns long-term view-invariant shape dynamics of action using human pose model (HPM) based view-invariant features mined from structural similarity index matrix (SSIM) based key depth human pose fram…
MIFI: MultI-camera Feature Integration for Roust 3D Distracted Driver Activity Recognition
High Magnetic Field Laboratory, HFIPS, Chinese Academy of Sciences（中国科学院合肥物质科学研究院强磁场科学中心）；University of Science and Technology of China（中国科学技术大学）；High Magnetic Field Laboratory of Anhui Province（安徽省强磁场实验室）
We investigate different approaches to integrating object recognition and planning in a tabletop manipulation domain with the set of objects used in the 2012 RoboCup@Work competition. Results of our preliminary experiments show that, with some approaches, close integration of perception and planning improves the quality of plans, as well as the computation times of feasible plans.
National Center for Mathematics and Interdisciplinary Sciences, Academy of Mathematics and Systems Science, CAS（中国科学院数学与系统科学研究院国家数学与交叉科学中心）；School of Mathematics Sciences, University of Chinese Academy of Sciences（中国科学院大学数学科学学院）
Nonnegative matrix factorization (NMF) is a powerful tool in data exploratory analysis by discovering the hidden features and part-based patterns from high-dimensional data. NMF and its variants have been successfully applied into diverse fields such as pattern recognition, signal processing, data mining, bioinformatics and so on. Recently, NMF has been extended to analyze multiple matrices simultaneously. However, a unified framework is still lacking. In this paper, we introduce a sparse multiple relationship data regularized joint matrix factorization (JMF) framework and two adapted predicti…
EPIR: An Efficient Patch Tokenization, Integration and Representation Framework for Micro-expression Recognition
School of Software, Northwestern Polytechnical University（西北工业大学软件学院）；School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；School of Computer Science and Technology, Xi'an Jiaotong University（西安交通大学计算机科学与技术学院）；School of Science, Edith Cowan University（埃迪斯科文大学理学院）
Long-Tailed Visual Recognition via Self-Heterogeneous Integration with Knowledge Excavation
Xiamen University（厦门大学）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））；Hong Kong Baptist University（香港浸会大学）
Inferring High-level Geographical Concepts via Knowledge Graph and Multi-scale Data Integration: A Case Study of C-shaped Building Pattern Recognition
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；School of Resource and Environment Science, Wuhan University（武汉大学资源与环境科学学院）；School of Information and Communication Engineering, Beijing University of Posts and Telecommunications（北京邮电大学信息与通信工程学院）；Institute of Remote Sensing and GIS, Peking University（北京大学遥感与地理信息系统研究所）
To fully understand the complexities of human emotion, the integration of multiple physical features from different modalities can be advantageous. Considering this, we present an analysis of 3D facial data, action units, and physiological data as it relates to their impact on emotion recognition. We analyze each modality independently, as well as the fusion of each for recognizing human emotion. This analysis includes which features are most important for specific emotions (e.g. happy). Our analysis indicates that both 3D facial landmarks and physiological data are encouraging for expression/…
Georgia Institute of Technology（佐治亚理工学院）；University of Sao Paulo（圣保罗大学）
The vehicle recognition area, including vehicle make-model recognition (VMMR), re-id, tracking, and parts-detection, has made significant progress in recent years, driven by several large-scale datasets for each task. These datasets are often non-overlapping, with different label schemas for each task: VMMR focuses on make and model, while re-id focuses on vehicle ID. It is promising to combine these datasets to take advantage of knowledge across datasets as well as increased training data; however, dataset integration is challenging due to the domain gap problem. This paper proposes ATEAM, an…
We propose a method for integration of features extracted using deep representations of Convolutional Neural Networks (CNNs) each of which is learned using a different image dataset of objects and materials for material recognition. Given a set of representations of multiple pre-trained CNNs, we first compute activations of features using the representations on the images to select a set of samples which are best represented by the features. Then, we measure the uncertainty of the features by computing the entropy of class distributions for each sample set. Finally, we compute the contribution…
Michigan State University（密歇根州立大学）；Ford Research and Innovation Center（福特研究与创新中心）
Gait, the walking pattern of individuals, is one of the most important biometrics modalities. Most of the existing gait recognition methods take silhouettes or articulated body models as the gait features. These methods suffer from degraded recognition performance when handling confounding variables, such as clothing, carrying and view angle. To remedy this issue, we propose a novel AutoEncoder framework to explicitly disentangle pose and appearance features from RGB imagery and the LSTM-based integration of pose features over time produces the gait feature. In addition, we collect a Frontal-V…
Towards automatic smoke detector inspection: Recognition of the smoke detectors in industrial facilities and preparation for future drone integration
Brno University of Technology（布尔诺理工大学）；VSB - Technical University of Ostrava（俄斯特拉发技术大学）；University of Ostrava（俄斯特拉发大学）；Mendel University in Brno（布尔诺孟德尔大学）
University Politehnica of Bucharest（布加勒斯特理工大学）；Xperi Corporation（Xperi公司）
Emotion recognition has a pivotal role in affective computing and in human-computer interaction. The current technological developments lead to increased possibilities of collecting data about the emotional state of a person. In general, human perception regarding the emotion transmitted by a subject is based on vocal and visual information collected in the first seconds of interaction with the subject. As a consequence, the integration of verbal (i.e., speech) and non-verbal (i.e., image) information seems to be the preferred choice in most of the current approaches towards emotion recognitio…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Institute of Applied Mathematics, Academy of Mathematics and Systems Science, Chinese Academy of Science（中国科学院数学与系统科学研究院应用数学研究所）
Integration between biology and information science benefits both fields. Many related models have been proposed, such as computational visual cognition models, computational motor control models, integrations of both and so on. In general, the robustness and precision of recognition is one of the key problems for object recognition models. In this paper, inspired by features of human recognition process and their biological mechanisms, a new integrated and dynamic framework is proposed to mimic the semantic extraction, concept formation and feature re-selection in human visual processing. The…
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China（中国科学技术大学认知智能国家重点实验室）；School of Computer Science, Liupanshui Normal University（黎平师范学院计算机科学学院）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
Laboratoire ETIS UMR8051（ETIS实验室（UMR8051））；Université Paris Seine（巴黎塞纳大学）；ENSEA（法国国立高等航空航天学院）；CNRS（法国国家科学研究中心）；Institut VEDECOM（VEDECOM研究所）
This paper introduces a novel unsupervised neural network model for visual information encoding which aims to address the problem of large-scale visual localization. Inspired by the structure of the visual cortex, the model (namely HSD) alternates layers of topologic sparse coding and pooling to build a more compact code of visual information. Intended for visual place recognition (VPR) systems that use local descriptors, the impact of its integration in a bio-inpired model for self-localization (LPMP) is evaluated. Our experimental results on the KITTI dataset show that HSD improves the runti…
Automatic Feature Recognition and Dimensional Attributes Extraction From CAD Models for Hybrid Additive-Subtractive Manufacturing
Singapore Institute of Manufacturing Technology (SIMTech), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局（A*STAR）新加坡制造技术研究院（SIMTech））；Advanced Remanufacturing and Technology Centre (ARTC), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局（A*STAR）先进再制造与技术中心（ARTC））；School of Mechanical and Aerospace Engineering, Nanyang Technological University（南洋理工大学机械与宇航工程学院）
针对混合制造场景下传统CAD特征自动识别方法无法识别增材特征、尺寸提取不准的问题，提出基于Python Open Cascade构建增减材复合特征合成数据集，用HGCNN实现高精度特征识别与尺寸提取，助力CAD/CAPP/CAM集成与工艺规划优化。
