In video action recognition, transformers consistently reach state-of-the-art accuracy. However, many models are too heavyweight for the average researcher with limited hardware resources. In this work, we explore the limitations of video transformers for lightweight action recognition. We benchmark 13 video transformers and baselines across 3 large-scale datasets and 10 hardware devices. Our study is the first to evaluate the efficiency of action recognition models in depth across multiple devices and train a wide range of video transformers under the same conditions. We categorize current methods into three classes and show that composite transformers that augment convolutional backbones are best at lightweight action recognition, despite lacking accuracy. Meanwhile, attention-only models need more motion modeling capabilities and stand-alone attention block models currently incur too much latency overhead. Our experiments conclude that current video transformers are not yet capable of lightweight action recognition on par with traditional convolutional baselines, and that the previously mentioned shortcomings need to be addressed to bridge this gap. Code to reproduce our experiments will be made publicly available.
With an increasing number of elders living alone, care-giving from a distance becomes a compelling need, particularly for safety. Real-time monitoring and action recognition are essential to raise an alert timely when abnormal behaviors or unusual activities occur. While wearable sensors are widely recognized as a promising solution, highly depending on user's ability and willingness makes them inefficient. In contrast, video streams collected through non-contact optical cameras provide richer information and release the burden on elders. In this paper, leveraging the Independently-Recurrent n…
Growing amount of different practical tasks in a video understanding problem has addressed the great challenge aiming to design an universal solution, which should be available for broad masses and suitable for the demanding edge-oriented inference. In this paper we are focused on designing a network architecture and a training pipeline to tackle the mentioned challenges. Our architecture takes the best from the previous ones and brings the ability to be successful not only in appearance-based action recognition tasks but in motion-based problems too. Furthermore, the induced label noise probl…
University of Edinburgh（爱丁堡大学）；Facebook AI Research（Facebook人工智能研究院）
Zero-shot action recognition is the task of classifying action categories that are not available in the training set. In this setting, the standard evaluation protocol is to use existing action recognition datasets(e.g. UCF101) and randomly split the classes into seen and unseen. However, most recent work builds on representations pre-trained on the Kinetics dataset, where classes largely overlap with classes in the zero-shot evaluation datasets. As a result, classes which are supposed to be unseen, are present during supervised pre-training, invalidating the condition of the zero-shot setting…
Efficient video action recognition remains a challenging problem. One large model after another takes the place of the state-of-the-art on the Kinetics dataset, but real-world efficiency evaluations are often lacking. In this work, we fill this gap and investigate the use of transformers for efficient action recognition. We propose a novel, lightweight action recognition architecture, VideoLightFormer. In a factorized fashion, we carefully extend the 2D convolutional Temporal Segment Network with transformers, while maintaining spatial and temporal video structure throughout the entire model. …
DVFL-Net: A Lightweight Distilled Video Focal Modulation Network for Spatio-Temporal Action Recognition
Intelligent Systems, Computer Architecture, Analytics, and Security Laboratory (ISCAAS Lab), Department of Electrical Engineering and Computer Science, Florida Atlantic University（智能系统、计算机架构、分析与安全实验室（ISCAAS实验室），电气工程与计算机科学系，佛罗里达 Atlantic 大学）；Intelligent Systems, Computer Architecture, Analytics, and Security Laboratory (ISCAAS Lab), Department of Electrical and Computer Engineering, Kansas State University（智能系统、计算机架构、分析与安全实验室（ISCAAS实验室），电气与计算机工程系，堪萨斯州立大学）
In this work we present a new efficient approach to Human Action Recognition called Video Transformer Network (VTN). It leverages the latest advances in Computer Vision and Natural Language Processing and applies them to video understanding. The proposed method allows us to create lightweight CNN models that achieve high accuracy and real-time speed using just an RGB mono camera and general purpose CPU. Furthermore, we explain how to improve accuracy by distilling from multiple models with different modalities into a single model. We conduct a comparison with state-of-the-art methods and show …
Action recognition is an important component to improve autonomy of physical rehabilitation devices, such as wearable robotic exoskeletons. Existing human action recognition algorithms focus on adult applications rather than pediatric ones. In this paper, we introduce BabyNet, a light-weight (in terms of trainable parameters) network structure to recognize infant reaching action from off-body stationary cameras. We develop an annotated dataset that includes diverse reaches performed while in a sitting posture by different infants in unconstrained environments (e.g., in home settings, etc.). Ou…
Physics-Guided Attention in a Lightweight TCN for Efficient WiFi CSI-Based Human Activity Recognition
Signal Processing, Artificial Intelligence and Vision Technologies (SAIVT) Research Group, School of Electrical Engineering and Robotics, Queensland University of Technology (QUT)（信号处理、人工智能与视觉技术（SAIVT）研究组，电气工程与机器人学院，昆士兰科技大学（QUT））
Sabaragamuwa University of Sri Lanka（斯里兰卡萨伯勒格穆沃大学）
Automatic tomato disease recognition from leaf images is vital to avoid crop losses by applying control measures on time. Even though recent deep learning-based tomato disease recognition methods with classical training procedures showed promising recognition results, they demand large labelled data and involve expensive training. The traditional deep learning models proposed for tomato disease recognition also consume high memory and storage because of a high number of parameters. While lightweight networks overcome some of these issues to a certain extent, they continue to show low performan…
Concordia University（康考迪亚大学）；University of Arkansas（阿肯色大学）；Carnegie Mellon University（卡内基梅隆大学）
Deep neural networks have been widely used in numerous computer vision applications, particularly in face recognition. However, deploying deep neural network face recognition on mobile devices has recently become a trend but still limited since most high-accuracy deep models are both time and GPU consumption in the inference stage. Therefore, developing a lightweight deep neural network is one of the most practical solutions to deploy face recognition on mobile devices. Such the lightweight deep neural network requires efficient memory with small number of weights representation and low cost o…
In the past decades the set of human tasks that are solved by machines was extended dramatically. From simple image classification problems researchers now move towards solving more sophisticated and vital problems, like, autonomous driving and language translation. The case of language translation includes a challenging area of sign language translation that incorporates both image and language processing. We make a step in that direction by proposing a lightweight network for ASL gesture recognition with a performance sufficient for practical applications. The proposed solution demonstrates …
Indian Institute of Technology, Kharagpur（印度理工学院克勒格布尔分校）
In this paper, we present a texture aware lightweight deep learning framework for iris recognition. Our contributions are primarily three fold. Firstly, to address the dearth of labelled iris data, we propose a reconstruction loss guided unsupervised pre-training stage followed by supervised refinement. This drives the network weights to focus on discriminative iris texture patterns. Next, we propose several texture aware improvisations inside a Convolution Neural Net to better leverage iris textures. Finally, we show that our systematic training and architectural choices enable us to design a…
In recent years, image recognition applications have developed rapidly. A large number of studies and techniques have emerged in different fields, such as face recognition, pedestrian and vehicle re-identification, landmark retrieval, and product recognition. In this paper, we propose a practical lightweight image recognition system, named PP-ShiTu, consisting of the following 3 modules, mainbody detection, feature extraction and vector search. We introduce popular strategies including metric learning, deep hash, knowledge distillation and model quantization to improve accuracy and inference s…
In this paper, we present an ultra lightweight system that can effectively recognize different circuit components in an image with very limited training data. Along with the system, we also release the data set we created for the task. A two-stage approach is employed by our system. Selective search was applied to find the location of each circuit component. Based on its result, we crop the original image into smaller pieces. The pieces are then fed to the Convolutional Neural Network (CNN) for classification to identify each circuit component. It is of engineering significance and works well …
Graph convolutional networks (GCNs) aim at extending deep learning to arbitrary irregular domains, namely graphs. Their success is highly dependent on how the topology of input graphs is defined and most of the existing GCN architectures rely on predefined or handcrafted graph structures. In this paper, we introduce a novel method that learns the topology (or connectivity) of input graphs as a part of GCN design. The main contribution of our method resides in building an orthogonal connectivity basis that optimally aggregates nodes, through their neighborhood, prior to achieve convolution. Our…
In this paper, the multi-task learning of lightweight convolutional neural networks is studied for face identification and classification of facial attributes (age, gender, ethnicity) trained on cropped faces without margins. The necessity to fine-tune these networks to predict facial expressions is highlighted. Several models are presented based on MobileNet, EfficientNet and RexNet architectures. It was experimentally demonstrated that they lead to near state-of-the-art results in age, gender and race recognition on the UTKFace dataset and emotion classification on the AffectNet dataset. Mor…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
With the development of convolutional neural network, significant progress has been made in computer vision tasks. However, the commonly used loss function softmax loss and highly efficient network architecture for common visual tasks are not as effective for face recognition. In this paper, we propose a novel loss function named Li-ArcFace based on ArcFace. Li-ArcFace takes the value of the angle through linear function as the target logit rather than through cosine function, which has better convergence and performance on low dimensional embedding feature learning for face recognition. In te…
University of Essex（埃塞克斯大学）；Queensland University of Technology（昆士兰科技大学）
In the last few years, Deep Convolutional Neural Networks (D-CNNs) have shown state-of-the-art (SOTA) performance for Visual Place Recognition (VPR), a pivotal component of long-term intelligent robotic vision (vision-aware localization and navigation systems). The prestigious generalization power of D-CNNs gained upon training on large scale places datasets and learned persistent image regions which are found to be robust for specific place recognition under changing conditions and camera viewpoints. However, against the computation and power intensive D-CNNs based VPR algorithms that are emp…
University of Essex（埃塞克斯大学）；Queensland University of Technology（昆士兰科技大学）
Low-overhead visual place recognition (VPR) is a highly active research topic. Mobile robotics applications often operate under low-end hardware, and even more hardware capable systems can still benefit from freeing up onboard system resources for other navigation tasks. This work addresses lightweight VPR by proposing a novel system based on the combination of binary-weighted classifier networks with a one-dimensional convolutional network, dubbed merger. Recent work in fusing multiple VPR techniques has mainly focused on increasing VPR performance, with computational efficiency not being hig…
HierCode: A Lightweight Hierarchical Codebook for Zero-shot Chinese Text Recognition
School of Electronic and Information Engineering, South China University of Technology, Guangzhou, China（华南理工大学电子与信息学院）；Alibaba DAMO Academy, Hangzhou, China（阿里巴巴达摩院）
To improve the discriminative and generalization ability of lightweight network for face recognition, we propose an efficient variable group convolutional network called VarGFaceNet. Variable group convolution is introduced by VarGNet to solve the conflict between small computational cost and the unbalance of computational intensity inside a block. We employ variable group convolution to design our network which can support large scale face identification while reduce computational cost and parameters. Specifically, we use a head setting to reserve essential information at the start of the net…
Mutual Modality Trust with Lightweight Reconstruction Regularization for Fine-grained Tire Pattern Recognition
School of Communications and Information Engineering, Xi’an University of Posts and Telecommunications（西安邮电大学通信与信息工程学院）；School of Artificial Intelligence, Hainan Normal University（海南师范大学人工智能学院）；School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University（西北工业大学人工智能与光电学院）
University of Arkansas（阿肯色大学）；Concordia University（康考迪亚大学）
Disentangled representations have been commonly adopted to Age-invariant Face Recognition (AiFR) tasks. However, these methods have reached some limitations with (1) the requirement of large-scale face recognition (FR) training data with age labels, which is limited in practice; (2) heavy deep network architectures for high performance; and (3) their evaluations are usually taken place on age-related face databases while neglecting the standard large-scale FR databases to guarantee robustness. This work presents a novel Lightweight Attentive Angular Distillation (LIAAD) approach to Large-scale…
Huazhong University of Science and Technology（华中科技大学）；Hubei University of Technology（湖北工业大学）
X-ray images are commonly used to ensure the security of devices in quality inspection industry. The recognition of signs printed on X-ray weld images plays an essential role in digital traceability system of manufacturing industry. However, the scales of objects vary different greatly in weld images, and it hinders us to achieve satisfactory recognition. In this paper, we propose a signs recognition framework based on convolutional neural networks (CNNs) for weld images. The proposed framework firstly contains a shallow classification network for correcting the pose of images. Moreover, we pr…
FilletRec: A Lightweight Graph Neural Network with Intrinsic Features for Automated Fillet Recognition
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；State Key Lab of CAD&CG, Zhejiang University（浙江大学计算机辅助设计与图形学国家重点实验室）；School of Aeronautics and Astronautics, Zhejiang University（浙江大学航空航天学院）
University of Essex（埃塞克斯大学）；ETH Zurich（苏黎世联邦理工学院）；Australian Centre for Robotic Vision（澳大利亚机器人视觉中心）；Queensland University of Technology（昆士兰科技大学）
This paper presents a lightweight visual place recognition approach, capable of achieving high performance with low computational cost, and feasible for mobile robotics under significant viewpoint and appearance changes. Results on several benchmark datasets confirm an average boost of 13% in accuracy, and 12x average speedup relative to state-of-the-art methods.
LPTR-AFLNet: Lightweight Integrated Chinese License Plate Rectification and Recognition Network
Hubei Key Laboratory of Intelligent Vision Based Monitoring for Hydroelectric Engineering, China Three Gorges University（湖北水利水电工程智能视觉监测重点实验室，中国三峡大学）；College of Computer and Information Technology, China Three Gorges University（计算机与信息学院，中国三峡大学）；Hubei Key Laboratory of Digital Finance Innovation（湖北省数字金融创新重点实验室）；School of Information Engineering, Hubei University of Economics（信息工程学院，湖北经济学院）
Fraunhofer Institute for Computer Graphics Research IGD（弗劳恩霍夫计算机图形学研究所）；TU Darmstadt（达姆施塔特工业大学）
Deep learning-based face recognition models follow the common trend in deep neural networks by utilizing full-precision floating-point networks with high computational costs. Deploying such networks in use-cases constrained by computational requirements is often infeasible due to the large memory required by the full-precision model. Previous compact face recognition approaches proposed to design special compact architectures and train them from scratch using real training data, which may not be available in a real-world scenario due to privacy concerns. We present in this work the QuantFace s…
Fraunhofer Institute for Computer Graphics Research IGD（弗劳恩霍夫计算机图形研究所）；Mathematical and Applied Visual Computing, TU Darmstadt（达姆施塔特工业大学数学与应用视觉计算）
Deep neural networks have rapidly become the mainstream method for face recognition (FR). However, this limits the deployment of such models that contain an extremely large number of parameters to embedded and low-end devices. In this work, we present an extremely lightweight and accurate FR solution, namely PocketNet. We utilize neural architecture search to develop a new family of lightweight face-specific architectures. We additionally propose a novel training paradigm based on knowledge distillation (KD), the multi-step KD, where the knowledge is distilled from the teacher model to the stu…
Image Recognition with Online Lightweight Vision Transformer: A Survey
Beijing University of Posts and Telecommunications（北京邮电大学）；State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center, Qilu University of Technology, Shandong Provincial Key Laboratory of Computing Power Internet and Service Computing, Shandong Fundamental Research Center for Computer Science（计算能力网络与信息安全重点实验室，教育部，山东计算机科学中心，齐鲁大学，山东省计算能力互联网与服务计算重点实验室，山东省计算机科学基础研究中心）
Hefei University of Technology（合肥工业大学）；Soochow University（苏州大学）；Harbin Institute of Technology（哈尔滨工业大学）；National University of Singapore（新加坡国立大学）
Convolutional Recurrent Neural Network (CRNN) is a popular network for recognizing texts in images. Advances like the variant of CRNN, such as Dense Convolutional Network with Connectionist Temporal Classification, has reduced the running time of the network, but exposing the inner computation cost and weight size of the convolutional networks as a bottleneck. Specifically, the DenseNet based models utilize the dense blocks as the core module, but the inner features are combined in the form of concatenation in dense blocks. As such, the number of channels of combined features delivered as the …
PAtt-Lite: Lightweight Patch and Attention MobileNet for Challenging Facial Expression Recognition
Faculty of Information Science and Technology, Multimedia University（多媒体大学信息科学与技术学院）；King Khalid University（哈立德国王大学）；Center for Artificial Intelligence (CAI), King Khalid University（哈立德国王大学人工智能中心（CAI））
WaveFormer: A Lightweight Transformer Model for sEMG-based Gesture Recognition
Integrated Systems Laboratory, ETH Zurich（苏黎世联邦理工学院集成系统实验室）；DEI, University of Bologna（博洛尼亚大学工程学院）；DIEF, University of Modena and Reggio Emilia（摩德纳和雷焦艾米利亚大学经济工程学院）
本文提出轻量级 Transformer 架构 WaveFormer，利用可学习小波变换和 WaveletConv 融合 sEMG 时频特征，以更少参数实现高准确率与 CPU 端低延迟实时手势识别。
Timely recognition of plant pests from field images is significant to avoid potential losses of crop yields. Traditional convolutional neural network-based deep learning models demand high computational capability and require large labelled samples for each pest type for training. On the other hand, the existing lightweight network-based approaches suffer in correctly classifying the pests because of common characteristics and high similarity between multiple plant pests. In this work, a novel double attention-based lightweight deep learning architecture is proposed to automatically recognize …
Human Activity Recognition (HAR) using wearable and mobile sensors has gained momentum in last few years, in various fields, such as, healthcare, surveillance, education, entertainment. Nowadays, Edge Computing has emerged to reduce communication latency and network traffic.Edge devices are resource constrained devices and cannot support high computation. In literature, various models have been developed for HAR. In recent years, deep learning algorithms have shown high performance in HAR, but these algorithms require lot of computation making them inefficient to be deployed on edge devices. T…
Concordia University（康考迪亚大学）；University of Arkansas（阿肯色大学）；Carnegie Mellon University（卡内基梅隆大学）
Large-scale face recognition in-the-wild has been recently achieved matured performance in many real work applications. However, such systems are built on GPU platforms and mostly deploy heavy deep network architectures. Given a high-performance heavy network as a teacher, this work presents a simple and elegant teacher-student learning paradigm, namely ShrinkTeaNet, to train a portable student network that has significantly fewer parameters and competitive accuracy against the teacher network. Far apart from prior teacher-student frameworks mainly focusing on accuracy and compression ratios i…
Evaluating Vision-Language Models as a Zero-Shot Learning Alternative to You Only Look Once and Optical Character Recognition for Nigerian License Plate Recognition
Ahmadu Bello University（艾哈迈杜·贝罗大学）；Bayero University Kano（卡诺贝罗大学）
