OAG-BERT: Towards A Unified Backbone Language Model For Academic Knowledge Services
Tsinghua University（清华大学）；National University of Singapore（新加坡国立大学）；Zhipu AI（智谱AI）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）
Xiao Liu, Da Yin, Jingnan Zheng, Xingjian Zhang, Peng Zhang, Hongxia Yang, Yuxiao Dong, Jie Tang
备注：
Accepted to KDD 2022
发表信息：
In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2022). Association for Computing Machinery, New York, NY, USA, 3418-3428
Academic knowledge services have substantially facilitated the development of the science enterprise by providing a plenitude of efficient research tools. However, many applications highly depend on ad-hoc models and expensive human labeling to understand scientific contents, hindering deployments into real products. To build a unified backbone language model for different knowledge-intensive academic applications, we pre-train an academic language model OAG-BERT that integrates both the heterogeneous entity knowledge and scientific corpora in the Open Academic Graph (OAG) -- the largest public academic graph to date. In OAG-BERT, we develop strategies for pre-training text and entity data along with zero-shot inference techniques. In OAG-BERT, we develop strategies for pre-training text and entity data along with zero-shot inference techniques. Its zero-shot capability furthers the path to mitigate the need of expensive annotations. OAG-BERT has been deployed for real-world applications, such as the reviewer recommendation function for National Nature Science Foundation of China (NSFC) -- one of the largest funding agencies in China -- and paper tagging in AMiner. All codes and pre-trained models are available via the CogDL toolkit.
Intel Labs（英特尔实验室）；Blavatnik School of Computer Science, Tel Aviv University（特拉维夫大学布拉瓦特尼克计算机科学学院）
While large language models a la BERT are used ubiquitously in NLP, pretraining them is considered a luxury that only a few well-funded industry labs can afford. How can one train such models with a more modest budget? We present a recipe for pretraining a masked language model in 24 hours using a single low-end deep learning server. We demonstrate that through a combination of software optimizations, design choices, and hyperparameter tuning, it is possible to produce models that are competitive with BERT-base on GLUE tasks at a fraction of the original pretraining cost.
University College Dublin（都柏林大学学院）；Birkbeck, University of London（伦敦大学伯贝克学院）
The contextual word embedding model, BERT, has proved its ability on downstream tasks with limited quantities of annotated data. BERT and its variants help to reduce the burden of complex annotation work in many interdisciplinary research areas, for example, legal argument mining in digital humanities. Argument mining aims to develop text analysis tools that can automatically retrieve arguments and identify relationships between argumentation clauses. Since argumentation is one of the key aspects of case law, argument mining tools for legal texts are applicable to both academic and non-academi…
Department of Computer Science；Engineering, Indian Institute of Technology, Kharagpur, West Bengal, India
Online hatred is a growing concern on many social media platforms. To address this issue, different social media platforms have introduced moderation policies for such content. They also employ moderators who can check the posts violating moderation policies and take appropriate action. Academicians in the abusive language research domain also perform various studies to detect such content better. Although there is extensive research in abusive language detection in English, there is a lacuna in abusive language detection in low resource languages like Hindi, Urdu etc. In this FIRE 2021 shared…
University of Toronto（多伦多大学）；Vector Institute（向量研究院）
Recently, large scale Transformer-based language models such as BERT, GPT-2, and XLNet have brought about exciting leaps in state-of-the-art results for many Natural Language Processing (NLP) tasks. One of the common trends in these recent models is a significant increase in model complexity, which introduces both more weights and computation. Moreover, with the advent of large-scale unsupervised datasets, training time is further extended due to the increased amount of data samples within a single training epoch. As a result, to train these models within a reasonable time, machine learning (M…
Contextualized entity representations learned by state-of-the-art transformer-based language models (TLMs) like BERT, GPT, T5, etc., leverage the attention mechanism to learn the data context from training data corpus. However, these models do not use the knowledge context. Knowledge context can be understood as semantics about entities and their relationship with neighboring entities in knowledge graphs. We propose a novel and effective technique to infuse knowledge context from multiple knowledge graphs for conceptual and ambiguous entities into TLMs during fine-tuning. It projects knowledge…
Detecting Redundant Health Survey Questions Using Language-agnostic BERT Sentence Embedding (LaBSE)
College of Natural Sciences, Seoul National University（首尔大学自然科学学院）；College of Nursing, Seoul National University（首尔大学护理学院）；The Research Institute of Nursing Science, Seoul National University（首尔大学护理科学研究所）；Samsung Medical Center（三星医疗中心）；David Geffen School of Medicine, University of California Los Angeles（加州大学洛杉矶分校大卫·格芬医学院）
Advancing Scientific Text Classification: Fine-Tuned Models with Dataset Expansion and Hard-Voting
Doctoral School of Applied Informatics（应用信息学与应用数学博士学院）；Applied Mathematics Obuda University Budapest, Hungary（应用数学 奥布达大学 布达佩斯 匈牙利）；John von Neumann Faculty of Informatics（约翰·冯·诺伊曼信息学学院）
Decoding Student Minds: Leveraging Conversational Agents for Psychological and Learning Analysis
STIH Laboratory, Sorbonne University（索邦大学STIH实验室）；LIPAH-LR11ES14, National Engineering School of Tunis（突尼斯国家工程学院LIPAH-LR11ES14）；Faculty of Sciences of Tunis（突尼斯科学学院）；LINFI Laboratory（LINFI实验室）
Nanjing University of Science and Technology（南京理工大学）；Nanjing Agricultural University（南京农业大学）；Wuhan University（武汉大学）
The academic literature of social sciences records human civilization and studies human social problems. With its large-scale growth, the ways to quickly find existing research on relevant issues have become an urgent demand for researchers. Previous studies, such as SciBERT, have shown that pre-training using domain-specific texts can improve the performance of natural language processing tasks. However, the pre-trained language model for social sciences is not available so far. In light of this, the present research proposes a pre-trained model based on the abstracts published in the Social …
Academic case reports lack diversity: Assessing the presence and diversity of sociodemographic and behavioral factors related to Post COVID-19 Condition
Dalla Lana School of Public Health, University of Toronto（多伦多大学达拉纳公共卫生学院）；York University（约克大学）；Vector Institute（矢量研究院）；Public Health Ontario（安大略省公共卫生局）
