Automatic emotion recognition (AER) is a challenging task due to the abstract concept and multiple expressions of emotion. Although there is no consensus on a definition, human emotional states usually can be apperceived by auditory and visual systems. Inspired by this cognitive process in human beings, it's natural to simultaneously utilize audio and visual information in AER. However, most traditional fusion approaches only build a linear paradigm, such as feature concatenation and multi-system fusion, which hardly captures complex association between audio and video. In this paper, we introduce factorized bilinear pooling (FBP) to deeply integrate the features of audio and video. Specifically, the features are selected through the embedded attention mechanism from respective modalities to obtain the emotion-related regions. The whole pipeline can be completed in a neural network. Validated on the AFEW database of the audio-video sub-challenge in EmotiW2018, the proposed approach achieves an accuracy of 62.48%, outperforming the state-of-the-art result.
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Princeton University（普林斯顿大学）
Discretization-based methods have been proposed for solving nonconvex optimization problems with bilinear terms such as the pooling problem. These methods convert the original nonconvex optimization problems into mixed-integer linear programs (MILPs). In this paper we study tightening methods for these MILP models for the pooling problem, and derive valid constraints using upper bounds on bilinear terms. Computational results demonstrate the effectiveness of our methods in terms of reducing solution time.
University of Minnesota（明尼苏达大学）；Samsung Research America（三星美国研究院）
Recommending new items to existing users has remained a challenging problem due to absence of user's past preferences for these items. The user personalized non-collaborative methods based on item features can be used to address this item cold-start problem. These methods rely on similarities between the target item and user's previous preferred items. While computing similarities based on item features, these methods overlook the interactions among the features of the items and consider them independently. Modeling interactions among features can be helpful as some features, when considered t…
Australian National University（澳大利亚国立大学）；Data61/CSIRO（Data61/联邦科学与工业研究组织）
Although Graph Convolutional Networks (GCNs) have demonstrated their power in various applications, the graph convolutional layers, as the most important component of GCN, are still using linear transformations and a simple pooling step. In this paper, we propose a novel generalization of Factorized Bilinear (FB) layer to model the feature interactions in GCNs. FB performs two matrix-vector multiplications, that is, the weight matrix is multiplied with the outer product of the vector of hidden features from both sides. However, the FB layer suffers from the quadratic number of coefficients, ov…
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；Saarland University（萨尔大学）
Knowledge graphs are incomplete by nature, with only a limited number of observed facts from the world knowledge being represented as structured relations between entities. To partly address this issue, an important task in statistical relational learning is that of link prediction or knowledge graph completion. Both linear and non-linear models have been proposed to solve the problem. Bilinear models, while expressive, are prone to overfitting and lead to quadratic growth of parameters in number of relations. Simpler models have become more standard, with certain constraints on bilinear map a…
Insight Centre for Data Analytics（洞察数据分析中心）；Dublin City University（都柏林城市大学）
Current deep learning based video classification architectures are typically trained end-to-end on large volumes of data and require extensive computational resources. This paper aims to exploit audio-visual information in video classification with a 1 frame per second sampling rate. We propose Temporal Bilinear Encoding Networks (TBEN) for encoding both audio and visual long range temporal information using bilinear pooling and demonstrate bilinear pooling is better than average pooling on the temporal dimension for videos with low sampling rate. We also embed the label hierarchy in TBEN to f…
Significant progress has recently been made in speaker diarisation after the introduction of d-vectors as speaker embeddings extracted from neural network (NN) speaker classifiers for clustering speech segments. To extract better-performing and more robust speaker embeddings, this paper proposes a c-vector method by combining multiple sets of complementary d-vectors derived from systems with different NN components. Three structures are used to implement the c-vectors, namely 2D self-attentive, gated additive, and bilinear pooling structures, relying on attention mechanisms, a gating mechanism…
The current methodology in tackling Acoustic Scene Classification (ASC) task can be described in two steps, preprocessing of the audio waveform into log-mel spectrogram and then using it as the input representation for Convolutional Neural Network (CNN). This paradigm shift occurs after DCASE 2016 where this framework model achieves the state-of-the-art result in ASC tasks on the (ESC-50) dataset and achieved an accuracy of 64.5%, which constitute to 20.5% improvement over the baseline model, and DCASE 2016 dataset with an accuracy of 90.0% (development) and 86.2% (evaluation), which constitut…
COMSATS University Islamabad（伊斯兰堡COMSATS大学）；Friedrich-Alexander University (FAU)（弗里德里希-亚历山大大学（FAU））；Data61-CSIRO（Data61-联邦科学与工业研究组织）；The Australian National University（澳大利亚国立大学）；TU Wien（维也纳工业大学）
We perform the classification of ancient Roman Republican coins via recognizing their reverse motifs where various objects, faces, scenes, animals, and buildings are minted along with legends. Most of these coins are eroded due to their age and varying degrees of preservation, thereby affecting their informative attributes for visual recognition. Changes in the positions of principal symbols on the reverse motifs also cause huge variations among the coin types. Lastly, in-plane orientations, uneven illumination, and a moderate background clutter further make the classification task non-trivial…
SK T-Brain；Seoul National University（首尔国立大学）；Surromind Robotics（索罗明德机器人公司）
Attention networks in multimodal learning provide an efficient way to utilize given visual information selectively. However, the computational cost to learn attention distributions for every pair of multimodal input channels is prohibitively expensive. To solve this problem, co-attention builds two separate attention distributions for each modality neglecting the interaction between multimodal inputs. In this paper, we propose bilinear attention networks (BAN) that find bilinear attention distributions to utilize given vision-language information seamlessly. BAN considers bilinear interactions…
QuIC: A Quantum-Inspired Interaction Classifier for Revitalizing Shallow CNNs in Fine-Grained Recognition
Big Data Chung Yuan Christian University Taoyuan City, Taiwan（大数据 Chung Yuan Christian 大学 台中市 台湾）；Big Data Quantum Information Center Chung Yuan Christian University Taoyuan City, Taiwan（大数据 量子信息中心 Chung Yuan Christian 大学 台中市 台湾）
Multi-Modal Opinion Integration for Financial Sentiment Analysis using Cross-Modal Attention
College of Computing Georgia Institute of Technology Atlanta, USA（计算学院 佐治亚理工学院 美国亚特兰大）；College of Engineering University of Pennsylvania Philadelphia, USA（工程学院 宾夕法尼亚大学 美国费城）
Accelerating Inference for Multilayer Neural Networks with Quantum Computers
Department of Materials, University of Oxford（牛津大学材料系）；Mathematical Institute, University of Oxford（牛津大学数学研究所）；Quantum Motion（Quantum Motion公司）；Centre for Quantum Technologies, National University of Singapore（新加坡国立大学量子中心）；Department of Computer Science, National University of Singapore（新加坡国立大学计算机科学系）
The University of Sydney（悉尼大学）；Royal Prince Alfred Hospital（皇家阿尔弗雷德王子医院）
Epilepsy affects nearly 1% of the global population, of which two thirds can be treated by anti-epileptic drugs and a much lower percentage by surgery. Diagnostic procedures for epilepsy and monitoring are highly specialized and labour-intensive. The accuracy of the diagnosis is also complicated by overlapping medical symptoms, varying levels of experience and inter-observer variability among clinical professions. This paper proposes a novel hybrid bilinear deep learning network with an application in the clinical procedures of epilepsy classification diagnosis, where the use of surface electr…
Modern neural network modules which can significantly enhance the learning power usually add too much computational complexity to the original neural networks. In this paper, we pursue very efficient neural network modules which can significantly boost the learning power of deep convolutional neural networks with negligible extra computational cost. We first present both theoretically and experimentally that elementwise square operator has a potential to enhance the learning power of neural networks. Then, we design four types of lightweight modules with elementwise square operators, named as …
Exploring Diverse Methods in Visual Question Answering
University of Michigan（密歇根大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Chongqing University（重庆大学）；Columbia University（哥伦比亚大学）；University of Chicago（芝加哥大学）
