This paper extends the Spatial-Temporal Graph Convolutional Network (ST-GCN) for skeleton-based action recognition by introducing two novel modules, namely, the Graph Vertex Feature Encoder (GVFE) and the Dilated Hierarchical Temporal Convolutional Network (DH-TCN). On the one hand, the GVFE module learns appropriate vertex features for action recognition by encoding raw skeleton data into a new feature space. On the other hand, the DH-TCN module is capable of capturing both short-term and long-term temporal dependencies using a hierarchical dilated convolutional network. Experiments have been conducted on the challenging NTU RGB-D-60 and NTU RGB-D 120 datasets. The obtained results show that our method competes with state-of-the-art approaches while using a smaller number of layers and parameters; thus reducing the required training time and memory.
Accelerating Streaming Video Large Language Models via Hierarchical Token Compression
EPIC Lab, Shanghai Jiao Tong University（上海交通大学EPIC实验室）；Sichuan University（四川大学）；Huazhong University of Science and Technology（华中科技大学）；Sun Yat-sen University（中山大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）；Centre for Frontier AI Research, A*STAR（新加坡科技研究局前沿人工智能研究中心）；Alibaba Group（阿里巴巴集团）
In this paper we present SA-CNN, a hierarchical and lightweight self-attention based encoding and decoding architecture for representation learning of point cloud data. The proposed SA-CNN introduces convolution and transposed convolution stacks to capture and generate contextual information among unordered 3D points. Following conventional hierarchical pipeline, the encoding process extracts feature in local-to-global manner, while the decoding process generates feature and point cloud in coarse-to-fine, multi-resolution stages. We demonstrate that SA-CNN is capable of a wide range of applica…
A biologically plausible computational model for color representation is introduced. We present a mechanistic hierarchical model of neurons that not only successfully encodes local hue, but also explicitly reveals how the contributions of each visual cortical layer participating in the process can lead to a hue representation. Our proposed model benefits from studies on the visual cortex and builds a network of single-opponent and hue-selective neurons. Local hue encoding is achieved through gradually increasing nonlinearity in terms of cone inputs to single-opponent cells. We demonstrate that…
Leiden University Medical Center（莱顿大学医学中心）；Philips Research Hamburg（飞利浦汉堡研究院）；Delft University of Technology（代尔夫特理工大学）
In Magnetic Resonance Fingerprinting (MRF) the quality of the estimated parameter maps depends on the encoding capability of the variable flip angle train. In this work we show how the dimensionality reduction technique Hierarchical Stochastic Neighbor Embedding (HSNE) can be used to obtain insight into the encoding capability of different MRF sequences. Embedding high-dimensional MRF dictionaries into a lower-dimensional space and visualizing them with colors, being a surrogate for location in low-dimensional space, provides a comprehensive overview of particular dictionaries and, in addition…
HiPose: Hierarchical Binary Surface Encoding and Correspondence Pruning for RGB-D 6DoF Object Pose Estimation
Zhejiang University（浙江大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；RPTU Kaiserslautern-Landau（凯撒斯劳滕-兰道工业大学）；Technische Universität München（慕尼黑工业大学）；Google（谷歌公司）
Over the past decade, deep neural networks have proven to be adept in image classification tasks, often surpassing humans in terms of accuracy. However, standard neural networks often fail to understand the concept of hierarchical structures and dependencies among different classes for vision related tasks. Humans on the other hand, seem to intuitively learn categories conceptually, progressively growing from understanding high-level concepts down to granular levels of categories. One of the issues arising from the inability of neural networks to encode such dependencies within its learned str…
Nanchang Hangkong University（南昌航空大学）；Dalian University of Technology（大连理工大学）；Cardiff University（卡迪夫大学）
Most of the existing audio-driven 3D facial animation methods suffered from the lack of detailed facial expression and head pose, resulting in unsatisfactory experience of human-robot interaction. In this paper, a novel pose-controllable 3D facial animation synthesis method is proposed by utilizing hierarchical audio-vertex attention. To synthesize real and detailed expression, a hierarchical decomposition strategy is proposed to encode the audio signal into both a global latent feature and a local vertex-wise control feature. Then the local and global audio features combined with vertex spati…
HierCode: A Lightweight Hierarchical Codebook for Zero-shot Chinese Text Recognition
School of Electronic and Information Engineering, South China University of Technology, Guangzhou, China（华南理工大学电子与信息学院）；Alibaba DAMO Academy, Hangzhou, China（阿里巴巴达摩院）
University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）
The use of Recurrent Neural Networks for video captioning has recently gained a lot of attention, since they can be used both to encode the input video and to generate the corresponding description. In this paper, we present a recurrent video encoding scheme which can discover and leverage the hierarchical structure of the video. Unlike the classical encoder-decoder approach, in which a video is encoded continuously by a recurrent layer, we propose a novel LSTM cell, which can identify discontinuity points between frames or segments and modify the temporal connections of the encoding layer acc…
MURE: Hierarchical Multi-Resolution Encoding via Vision-Language Models for Visual Document Retrieval
National University of Singapore, Singapore（新加坡国立大学）；University of Science and Technology of China, China（中国科学技术大学）；Hefei University of Technology, China（合肥工业大学）
We address the task of domain generalization, where the goal is to train a predictive model such that it is able to generalize to a new, previously unseen domain. We choose a hierarchical generative approach within the framework of variational autoencoders and propose a domain-unsupervised algorithm that is able to generalize to new domains without domain supervision. We show that our method is able to learn representations that disentangle domain-specific information from class-label specific information even in complex settings where domain structure is not observed during training. Our inte…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Netflix Inc.（Netflix公司）
In VP9 video codec, the sizes of blocks are decided during encoding by recursively partitioning 64$\times$64 superblocks using rate-distortion optimization (RDO). This process is computationally intensive because of the combinatorial search space of possible partitions of a superblock. Here, we propose a deep learning based alternative framework to predict the intra-mode superblock partitions in the form of a four-level partition tree, using a hierarchical fully convolutional network (H-FCN). We created a large database of VP9 superblocks and the corresponding partitions to train an H-FCN mode…
In this work, we present novel temporal encoding methods for action and activity classification by extending the unsupervised rank pooling temporal encoding method in two ways. First, we present "discriminative rank pooling" in which the shared weights of our video representation and the parameters of the action classifiers are estimated jointly for a given training dataset of labelled vector sequences using a bilevel optimization formulation of the learning problem. When the frame level features vectors are obtained from a convolutional neural network (CNN), we rank pool the network activatio…
Hunan University（湖南大学）；The University of Western Australia（西澳大学）
Point cloud processing methods exploit local point features and global context through aggregation which does not explicity model the internal correlations between local and global features. To address this problem, we propose full point encoding which is applicable to convolution and transformer architectures. Specifically, we propose Full Point Convolution (FPConv) and Full Point Transformer (FPTransformer) architectures. The key idea is to adaptively learn the weights from local and global geometric connections, where the connections are established through local and global correlation func…
Technical University of Munich（慕尼黑工业大学）；Alibaba Inc.（阿里巴巴集团）；Simon Fraser University（西蒙弗雷泽大学）
Spotting graphical symbols from the computer-aided design (CAD) drawings is essential to many industrial applications. Different from raster images, CAD drawings are vector graphics consisting of geometric primitives such as segments, arcs, and circles. By treating each CAD drawing as a graph, we propose a novel graph attention network GAT-CADNet to solve the panoptic symbol spotting problem: vertex features derived from the GAT branch are mapped to semantic labels, while their attention scores are cascaded and mapped to instance prediction. Our key contributions are three-fold: 1) the instanc…
HiDiGen: Hierarchical Diffusion for B-Rep Generation with Explicit Topological Constraints
School of Computer Science and Engineering, Sun Yat-sen University, China（中山大学计算机科学与工程学院）；School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University, China（中山大学深圳校区智能工程学院）
Indian Institute of Technology Gandhinagar（印度理工学院甘地纳加尔分校）
Point cloud is one of the widely used techniques for representing and storing 3D geometric data. In the past several methods have been proposed for processing point clouds. Methods such as PointNet and FoldingNet have shown promising results for tasks like 3D shape classification and segmentation. This work proposes a tree-structured autoencoder framework to generate robust embeddings of point clouds by utilizing hierarchical information using graph convolution. We perform multiple experiments to assess the quality of embeddings generated by the proposed encoder architecture and visualize the …
Academy of information systems engineering, PLA strategy support force information engineering university（中国人民解放军战略支援部队信息工程大学信息系统工程学院）
On basis of functional magnetic resonance imaging (fMRI), researchers are devoted to designing visual encoding models to predict the neuron activity of human in response to presented image stimuli and analyze inner mechanism of human visual cortices. Deep network structure composed of hierarchical processing layers forms deep network models by learning features of data on specific task through big dataset. Deep network models have powerful and hierarchical representation of data, and have brought about breakthroughs for visual encoding, while revealing hierarchical structural similarity with t…
We present a new permutation-invariant network for 3D point cloud processing. Our network is composed of a recurrent set encoder and a convolutional feature aggregator. Given an unordered point set, the encoder firstly partitions its ambient space into parallel beams. Points within each beam are then modeled as a sequence and encoded into subregional geometric features by a shared recurrent neural network (RNN). The spatial layout of the beams is regular, and this allows the beam features to be further fed into an efficient 2D convolutional neural network (CNN) for hierarchical feature aggrega…
HiPath: Hierarchical Vision-Language Alignment for Structured Pathology Report Prediction
College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机科学与电子工程学院）；Department of Bioengineering and Imperial-X, Imperial College London（帝国理工学院伦敦校区生物工程系）；Department of Pathology, Xiangtan Maternal and Child Health Hospital（湘潭 maternal and child health hospital pathology department）；Department of Pathology, The First People’s Hospital of Xiangtan City（湘潭市第一人民医院病理科）
