Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning
Center for Artificial Intelligence Research (CAiRE)（人工智能研究中心）；Department of Electronic and Computer Engineering（电子与计算机工程系）；The Hong Kong University of Science and Technology（香港科技大学）
Zhaojiang Lin, Andrea Madotto, Pascale Fung
备注：
Accepted as Findings of EMNLP 2020, Zhaojiang Lin and Andrea Madotto contributed equally to this work
Fine-tuning pre-trained generative language models to down-stream language generation tasks has shown promising results. However, this comes with the cost of having a single, large model for each task, which is not ideal in low-memory/power scenarios (e.g., mobile). In this paper, we propose an effective way to fine-tune multiple down-stream generation tasks simultaneously using a single, large pre-trained model. The experiments on five diverse language generation tasks show that by just using an additional 2-3% parameters for each task, our model can maintain or even improve the performance of fine-tuning the whole model.
The Chinese University of Hong Kong（香港中文大学）；Tencent AI Lab（腾讯人工智能实验室）；Alibaba DAMO Academy（阿里巴巴达摩院）
In this paper, we present a novel integrated approach for keyphrase generation (KG). Unlike previous works which are purely extractive or generative, we first propose a new multi-task learning framework that jointly learns an extractive model and a generative model. Besides extracting keyphrases, the output of the extractive model is also employed to rectify the copy probability distribution of the generative model, such that the generative model can better identify important contents from the given document. Moreover, we retrieve similar documents with the given document from training data an…
Attribute values of the products are an essential component in any e-commerce platform. Attribute Value Extraction (AVE) deals with extracting the attributes of a product and their values from its title or description. In this paper, we propose to tackle the AVE task using generative frameworks. We present two types of generative paradigms, namely, word sequence-based and positional sequence-based, by formulating the AVE task as a generation problem. We conduct experiments on two datasets where the generative approaches achieve the new state-of-the-art results. This shows that we can use the p…
The University of Tokyo（东京大学）；National Institute of Advanced Industrial Science and Technology (AIST)（产业技术综合研究所）
Transformers have shown improved performance when compared to previous architectures for sequence processing such as RNNs. Despite their sizeable performance gains, as recently suggested, the model is computationally expensive to train and with a high parameter budget. In light of this, we explore parameter-sharing methods in Transformers with a specific focus on generative models. We perform an analysis of different parameter sharing/reduction methods and develop the Subformer. Our model combines sandwich-style parameter sharing, which overcomes naive cross-layer parameter sharing in generati…
Exploring Generative Process Reward Modeling for Semi-Structured Data: A Case Study of Table Question Answering
University of Augsburg（奥格斯堡大学）；Institut für Maschinelle Sprachverarbeitung, University of Stuttgart（斯图加特大学机械语言处理研究所）；Bosch Center for Artificial Intelligence（博世人工智能中心）
Exploring Training and Inference Scaling Laws in Generative Retrieval
National University of Singapore（新加坡国立大学）；The Hong Kong Polytechnic University（香港理工大学）；University of Science and Technology of China（中国科学技术大学）；Nanyang Technological University（南洋理工大学）
Exploring LGBTQ+ Bias in Generative AI Answers across Different Country and Religious Contexts
Corvinus University of Budapest（布达佩斯考文纽斯大学）；Central European University Democracy Institute（中欧大学民主研究所）；University of British Columbia Okanagan（不列颠哥伦比亚大学奥肯那根分校）；HUN-REN Centre for Social Sciences（匈牙利研究网络社会科学中心）
Jilin University（吉林大学）；Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University（吉林大学符号计算与知识工程教育部重点实验室）
Dataless text classification, i.e., a new paradigm of weakly supervised learning, refers to the task of learning with unlabeled documents and a few predefined representative words of categories, known as seed words. The recent generative dataless methods construct document-specific category priors by using seed word occurrences only, however, such category priors often contain very limited and even noisy supervised signals. To remedy this problem, in this paper we propose a novel formulation of category prior. First, for each document, we consider its label membership degree by not only counti…
Exploring the Use of Abusive Generative AI Models on Civitai
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；Queen Mary University of London（伦敦大学玛丽女王学院）；University of Helsinki（赫尔辛基大学）
