Voluminous works have been implemented to exploit content-enhanced network embedding models, with little focus on the labelled information of nodes. Although TriDNR leverages node labels by treating them as node attributes, it fails to enrich unlabelled node vectors with the labelled information, which leads to the weaker classification result on the test set in comparison to existing unsupervised textual network embedding models. In this study, we design an integrated node encoder (INE) for textual networks which is jointly trained on the structure-based and label-based objectives. As a result, the node encoder preserves the integrated knowledge of not only the network text and structure, but also the labelled information. Furthermore, INE allows the creation of label-enhanced vectors for unlabelled nodes by entering their node contents. Our node embedding achieves state-of-the-art performances in the classification task on two public citation networks, namely Cora and DBLP, pushing benchmarks up by 10.0\% and 12.1\%, respectively, with the 70\% training ratio. Additionally, a feasible solution that generalizes our model from textual networks to a broader range of networks is proposed.
University of Toronto（多伦多大学）；Vector Institute for Artificial Intelligence（向量人工智能研究所）；University Health Network（大学健康网络）
Sentence embeddings are an important component of many natural language processing (NLP) systems. Like word embeddings, sentence embeddings are typically learned on large text corpora and then transferred to various downstream tasks, such as clustering and retrieval. Unlike word embeddings, the highest performing solutions for learning sentence embeddings require labelled data, limiting their usefulness to languages and domains where labelled data is abundant. In this paper, we present DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations. Inspired by recent advances in d…
The University of Queensland（昆士兰大学）；Monash University（莫纳什大学）
Current approaches which are mainly based on the extraction of low-level relations among individual events are limited by the shortage of publicly available labelled data. Therefore, the resulting models perform poorly when applied to a distributionally different domain for which labelled data did not exist at the time of training. To overcome this limitation, in this paper, we leverage the characteristics of dependency trees and adversarial learning to address the tasks of adaptive causality identification and localisation. The term adaptive is used since the training and test data come from …
Center for Information and Language Processing, LMU Munich（慕尼黑大学信息与语言处理中心）；Siemens AG（西门子股份公司）
We address the task of unsupervised Semantic Textual Similarity (STS) by ensembling diverse pre-trained sentence encoders into sentence meta-embeddings. We apply, extend and evaluate different meta-embedding methods from the word embedding literature at the sentence level, including dimensionality reduction (Yin and Schütze, 2016), generalized Canonical Correlation Analysis (Rastogi et al., 2015) and cross-view auto-encoders (Bollegala and Bao, 2018). Our sentence meta-embeddings set a new unsupervised State of The Art (SoTA) on the STS Benchmark and on the STS12-STS16 datasets, with gains of …
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Tencent Cloud Xiaowei（腾讯云小微）
Existing text- and image-based multimodal dialogue systems use the traditional Hierarchical Recurrent Encoder-Decoder (HRED) framework, which has an utterance-level encoder to model utterance representation and a context-level encoder to model context representation. Although pioneer efforts have shown promising performances, they still suffer from the following challenges: (1) the interaction between textual features and visual features is not fine-grained enough. (2) the context representation can not provide a complete representation for the context. To address the issues mentioned above, w…
CoT-MoTE: Exploring ConTextual Masked Auto-Encoder Pre-training with Mixture-of-Textual-Experts for Passage Retrieval
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Kuaishou Technology（快手科技）
University of Chinese Academy of Sciences（中国科学院大学）；Noah’s Ark Lab, Huawei（华为诺亚方舟实验室）；Huawei Technologies Ltd.（华为技术有限公司）
Recent advances in neural-based generative modeling have reignited the hopes of having computer systems capable of conversing with humans and able to understand natural language. The employment of deep neural architectures has been largely explored in a multitude of context and tasks to fulfill various user needs. On one hand, producing textual content that meets specific requirements is of priority for a model to seamlessly conduct conversations with different groups of people. On the other hand, latent variable models (LVM) such as variational auto-encoders (VAEs) as one of the most popular …
Northeastern University（东北大学）；Tencent（腾讯）；The Chinese University of Hong Kong（香港中文大学）；NiuTrans Research（小牛翻译研究院）
Encoder pre-training is promising in end-to-end Speech Translation (ST), given the fact that speech-to-translation data is scarce. But ST encoders are not simple instances of Automatic Speech Recognition (ASR) or Machine Translation (MT) encoders. For example, we find that ASR encoders lack the global context representation, which is necessary for translation, whereas MT encoders are not designed to deal with long but locally attentive acoustic sequences. In this work, we propose a Stacked Acoustic-and-Textual Encoding (SATE) method for speech translation. Our encoder begins with processing th…
Labelled network subgraphs reveal stylistic subtleties in written texts
Institute of Mathematics and Computer Science, University of São Paulo（圣保罗大学数学与计算机科学研究所）；Department of Computer Science, University of Toronto（多伦多大学计算机科学系）
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；International Center of Future Science, Jilin University（吉林大学未来科学国际中心）
Human-curated knowledge graphs provide critical supportive information to various natural language processing tasks, but these graphs are usually incomplete, urging auto-completion of them. Prevalent graph embedding approaches, e.g., TransE, learn structured knowledge via representing graph elements into dense embeddings and capturing their triple-level relationship with spatial distance. However, they are hardly generalizable to the elements never visited in training and are intrinsically vulnerable to graph incompleteness. In contrast, textual encoding approaches, e.g., KG-BERT, resort to gr…
Leveraging context information is an intuitive idea to improve performance on conversational automatic speech recognition(ASR). Previous works usually adopt recognized hypotheses of historical utterances as preceding context, which may bias the current recognized hypothesis due to the inevitable historicalrecognition errors. To avoid this problem, we propose an audio-textual cross-modal representation extractor to learn contextual representations directly from preceding speech. Specifically, it consists of two modal-related encoders, extracting high-level latent features from speech and the co…
We provide the first exploration of sentence embeddings from text-to-text transformers (T5). Sentence embeddings are broadly useful for language processing tasks. While T5 achieves impressive performance on language tasks cast as sequence-to-sequence mapping problems, it is unclear how to produce sentence embeddings from encoder-decoder models. We investigate three methods for extracting T5 sentence embeddings: two utilize only the T5 encoder and one uses the full T5 encoder-decoder model. To support our investigation, we establish a new sentence representation transfer benchmark, SentGLUE, wh…
Dublin City University（都柏林城市大学）；Westlake University（西湖大学）；Trinity College Dublin（都柏林圣三一学院）
User and product information associated with a review is useful for sentiment polarity prediction. Typical approaches incorporating such information focus on modeling users and products as implicitly learned representation vectors. Most do not exploit the potential of historical reviews, or those that currently do require unnecessary modifications to model architecture or do not make full use of user/product associations. The contribution of this work is twofold: i) a method to explicitly employ historical reviews belonging to the same user/product to initialize representations, and ii) effici…
Learning semantically meaningful sentence embeddings is an open problem in natural language processing. In this work, we propose a sentence embedding learning approach that exploits both visual and textual information via a multimodal contrastive objective. Through experiments on a variety of semantic textual similarity tasks, we demonstrate that our approach consistently improves the performance across various datasets and pre-trained encoders. In particular, combining a small amount of multimodal data with a large text-only corpus, we improve the state-of-the-art average Spearman's correlati…
SciTopic: Enhancing Topic Discovery in Scientific Literature through Advanced LLM
Computer Network Information Center, Chinese Academy of Sciences, Beijing, China（中国科学院计算机网络信息中心）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学）；Department of Computer Science, Portland State University, Portland, US（波特兰州立大学计算机科学系）；Information Science and Technology College, Dalian Maritime University, Dalian, China（大连海事大学信息科学与技术学院）
Learning sentence embeddings often requires a large amount of labeled data. However, for most tasks and domains, labeled data is seldom available and creating it is expensive. In this work, we present a new state-of-the-art unsupervised method based on pre-trained Transformers and Sequential Denoising Auto-Encoder (TSDAE) which outperforms previous approaches by up to 6.4 points. It can achieve up to 93.1% of the performance of in-domain supervised approaches. Further, we show that TSDAE is a strong domain adaptation and pre-training method for sentence embeddings, significantly outperforming …
Politecnico di Milano（米兰理工大学）；Università di Bologna（博洛尼亚大学）
Semantic sentence embeddings are usually supervisedly built minimizing distances between pairs of embeddings of sentences labelled as semantically similar by annotators. Since big labelled datasets are rare, in particular for non-English languages, and expensive, recent studies focus on unsupervised approaches that require not-paired input sentences. We instead propose a language-independent approach to build large datasets of pairs of informal texts weakly similar, without manual human effort, exploiting Twitter's intrinsic powerful signals of relatedness: replies and quotes of tweets. We use…
NAG: A Unified Native Architecture for Encoder-free Text-Graph Modeling in Language Models
New Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences, Beijing, China；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS), Beijing, China；School of Advanced Interdisciplinary Sciences, UCAS, Beijing, China
Embedding Biomedical Ontologies by Jointly Encoding Network Structure and Textual Node Descriptors
Athens University of Economics and Business（雅典经济与商业大学）；Institute for Language and Speech Processing, Research Center ‘Athena’（“雅典娜”研究中心语言与语音处理研究所）；Google Research（谷歌研究院）；CNRS（法国国家科学研究中心）；Univ. Paris Diderot（巴黎第七大学）
Open-domain extractive question answering works well on textual data by first retrieving candidate texts and then extracting the answer from those candidates. However, some questions cannot be answered by text alone but require information stored in tables. In this paper, we present an approach for retrieving both texts and tables relevant to a question by jointly encoding texts, tables and questions into a single vector space. To this end, we create a new multi-modal dataset based on text and table datasets from related work and compare the retrieval performance of different encoding schemata…
Conversational Speech Recognition by Learning Audio-textual Cross-modal Contextual Representation
School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；School of Computer Science and Engineering, Northeastern University（东北大学计算机科学与工程学院）；Mashang Consumer Finance Co., Ltd.（马上消费金融股份有限公司）
JD AI Research（京东人工智能研究院）；Sun Yat-sen University（中山大学）
Despite having impressive vision-language (VL) pretraining with BERT-based encoder for VL understanding, the pretraining of a universal encoder-decoder for both VL understanding and generation remains challenging. The difficulty originates from the inherently different peculiarities of the two disciplines, e.g., VL understanding tasks capitalize on the unrestricted message passing across modalities, while generation tasks only employ visual-to-textual message passing. In this paper, we start with a two-stream decoupled design of encoder-decoder structure, in which two decoupled cross-modal enc…
Leveraging Graph Structure in Seq2Seq Models for Knowledge Graph Link Prediction
Analytic Computing, KI, University of Stuttgart, Stuttgart, Germany（斯图加特大学分析计算研究所）；Bosch Center for Artificial Intelligence, Stuttgart, Germany（博世人工智能中心）；WAIS, University of Southampton, United Kingdom（南安普顿大学WAIS）
Text to Band Gap: Pre-trained Language Models as Encoders for Semiconductor Band Gap Prediction
Department of Chemical Engineering, Carnegie Mellon University（化学工程系，卡内基梅隆大学）；Department of Mechanical Engineering, Carnegie Mellon University（机械工程系，卡内基梅隆大学）；Department of Material Science Engineering, Carnegie Mellon University（材料科学工程系，卡内基梅隆大学）
Identifying multiple speakers without knowing where a speaker's voice is in a recording is a challenging task. In this paper, a hierarchical attention network is proposed to solve a weakly labelled speaker identification problem. The use of a hierarchical structure, consisting of a frame-level encoder and a segment-level encoder, aims to learn speaker related information locally and globally. Speech streams are segmented into fragments. The frame-level encoder with attention learns features and highlights the target related frames locally, and output a fragment based embedding. The segment-lev…
Self-supervised bidirectional transformer models such as BERT have led to dramatic improvements in a wide variety of textual classification tasks. The modern digital world is increasingly multimodal, however, and textual information is often accompanied by other modalities such as images. We introduce a supervised multimodal bitransformer model that fuses information from text and image encoders, and obtain state-of-the-art performance on various multimodal classification benchmark tasks, outperforming strong baselines, including on hard test sets specifically designed to measure multimodal pe…
