In natural language processing (NLP), the context of a word or sentence plays an essential role. Contextual information such as the semantic representation of a passage or historical dialogue forms an essential part of a conversation and a precise understanding of the present phrase or sentence. However, the standard attention mechanisms typically generate weights using query and key but ignore context, forming a Bi-Attention framework, despite their great success in modeling sequence alignment. This Bi-Attention mechanism does not explicitly model the interactions between the contexts, queries and keys of target sequences, missing important contextual information and resulting in poor attention performance. Accordingly, a novel and general triple-attention (Tri-Attention) framework expands the standard Bi-Attention mechanism and explicitly interacts query, key, and context by incorporating context as the third dimension in calculating relevance scores. Four variants of Tri-Attention are generated by expanding the two-dimensional vector-based additive, dot-product, scaled dot-product, and bilinear operations in Bi-Attention to the tensor operations for Tri-Attention. Extensive experiments on three NLP tasks demonstrate that Tri-Attention outperforms about 30 state-of-the-art non-attention, standard Bi-Attention, contextual Bi-Attention approaches and pretrained neural language models1.
TRACE: State-Aware Query Processing over Temporal Evidence Graphs for Conversational Data
Hong Kong Institute of AI for Science, City University of Hong Kong（香港城市大学香港人工智能科学研究院）；Independent Researcher（独立研究者）；Central South University（中南大学）；The Hong Kong Polytechnic University（香港理工大学）
Query-focused and Memory-aware Reranker for Long Context Processing
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；Pattern Recognition Center, WeChat AI, Tencent（腾讯微信人工智能研究院模式识别中心）；East China Normal University（华东师范大学）
Efficient Dialect-Aware Modeling and Conditioning for Low-Resource Taiwanese Hakka Speech Processing
Dept. Computer Science and Information Engineering, National Taiwan Normal University, Taiwan（计算机科学与信息工程系，国立台湾师范大学，台湾）；Institute of Computer Science, Academia Sinica, Taiwan（学术院计算机科学研究所，台湾）；United Link Co., Ltd., Taiwan（联合链接公司，台湾）
