语言大模型 / LLM

Language Models as a Knowledge Source for Cognitive Agents

Robert E. Wray, III, James R. Kirk, John E. Laird

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.AI

Comments 16 pages, 2 figures; accepted for 2021 Advances in Cognitive Systems Conference (revised based on reviews)

Knowledge distillation from language model to acoustic model: a hierarchical multi-task learning approach

Mun-Hak Lee, Joon-Hyuk Chang

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments 4page + 1page for citation + 2 pages for appendix

Maximizing Efficiency of Language Model Pre-training for Learning Representation

Junmo Kang, Suwon Shin, Jeonghwan Kim, Jaeyoung Jo, Sung-Hyon Myaeng

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments Published in KSC 2020

Language Modeling using LMUs: 10x Better Data Efficiency or Improved Scaling Compared to Transformers

Narsimha Chilkuri, Eric Hunsberger, Aaron Voelker, Gurshaant Malik, Chris Eliasmith

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.LG

Dynamic Knowledge Distillation for Pre-trained Language Models

Lei Li, Yankai Lin, Shuhuai Ren, Peng Li, Jie Zhou, Xu Sun

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments Main Conference EMNLP 2021, Camera Ready

A Discriminative Entity-Aware Language Model for Virtual Assistants

Mandana Saebi, Ernest Pusateri, Aaksha Meghawat, Christophe Van Gysel

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments To appear in Interspeech 2021

KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation

Marzieh S. Tahaei, Ella Charlaix, Vahid Partovi Nia, Ali Ghodsi, Mehdi Rezagholizadeh

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.AI

DEMix Layers: Disentangling Domains for Modular Language Modeling

Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, Luke Zettlemoyer

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.AI

Comments edits: updated reference links, added related work, typo fixes

AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models

Yichun Yin, Cheng Chen, Lifeng Shang, Xin Jiang, Xiao Chen, Qun Liu

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments ACL 2021. The code and models are released at https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/AutoTinyBERT

Do Language Models Perform Generalizable Commonsense Inference?

Peifeng Wang, Filip Ilievski, Muhao Chen, Xiang Ren

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.AI

Comments 8 pages, 4 figures. Accepted to ACL'21 Findings

Dynamic Language Models for Continuously Evolving Content

Spurthi Amba Hombaiah, Tao Chen, Mingyang Zhang, Michael Bendersky, Marc Najork

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.LG

Journal ref KDD 2021

Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization

Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, Weizhu Chen

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments The 59th annual meeting of the Association for Computational Linguistics (ACL 2021)

AMBERT: A Pre-trained Language Model with Multi-Grained Tokenization

Xinsong Zhang, Pengshuai Li, Hang Li

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments To be appeared in Findings of ACL2021. In this version, we develop a simplified method to improve the efficiency of AMBERT in inference, which still performs better than BERT with the same computational cost as BERT

Post-training deep neural network pruning via layer-wise calibration

Ivan Lazarevich, Alexander Kozlov, Nikita Malinin

专题命中 效率与部署 ：post-training(title,abstract)；分类 cs.AI、cs.LG

Language Models for Lexical Inference in Context

Martin Schmitt, Hinrich Schütze

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.AI

Comments Final version of EACL 2021 long paper

Data-Efficient Pretraining via Contrastive Self-Supervision

Nils Rethmeier, Isabelle Augenstein

专题命中 效率与部署 ：pretraining(title,abstract)；分类 cs.CL、cs.LG

Comments Majorly reworked version. Comparison to a large-scale RoBERTa model added. Focus on learning efficiency comparison to self and RoBERTa

Evaluating Post-Training Compression in GANs using Locality-Sensitive Hashing

Gonçalo Mordido, Haojin Yang, Christoph Meinel

专题命中 效率与部署 ：post-training(title,abstract)；分类 cs.AI、cs.LG

Training Large-Scale News Recommenders with Pretrained Language Models in the Loop

Shitao Xiao, Zheng Liu, Yingxia Shao, Tao Di, Xing Xie

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.AI

Automated essay scoring using efficient transformer-based language models

Christopher M Ormerod, Akanksha Malhotra, Amir Jafari

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments 11 pages, 1 figure, 3 tables

Evaluating Commonsense in Pre-trained Language Models

Xuhui Zhou, Yue Zhang, Leyang Cui, Dandan Huang

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.AI

Comments AAAI 2020

Masked Language Model Scoring

Julian Salazar, Davis Liang, Toan Q. Nguyen, Katrin Kirchhoff

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments ACL 2020 camera-ready (presented July 2020)

Journal ref Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020), 2699-2712

Scaling Hidden Markov Language Models

Justin T. Chiu, Alexander M. Rush

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments 9 pages, accepted as a short paper at EMNLP 2020

Journal ref EMNLP 2020

Real-Time Execution of Large-scale Language Models on Mobile

Wei Niu, Zhenglun Kong, Geng Yuan, Weiwen Jiang, Jiexiong Guan, Caiwen Ding, Pu Zhao, Sijia Liu, Bin Ren, Yanzhi Wang

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.LG

Improving Self-supervised Pre-training via a Fully-Explored Masked Language Model

Mingzhi Zheng, Dinghan Shen, Yelong Shen, Weizhu Chen, Lin Xiao

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.AI

PhoBERT: Pre-trained language models for Vietnamese

Dat Quoc Nguyen, Anh Tuan Nguyen

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2020 (Findings)

Contrastive Distillation on Intermediate Representations for Language Model Compression

Siqi Sun, Zhe Gan, Yu Cheng, Yuwei Fang, Shuohang Wang, Jingjing Liu

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments Accepted by EMNLP 2020

Multi-node Bert-pretraining: Cost-efficient Approach

Jiahuang Lin, Xin Li, Gennady Pekhimenko

专题命中 效率与部署 ：pretraining(title)；language model(abstract)；分类 cs.CL、cs.LG

MicroNet for Efficient Language Modeling

Zhongxia Yan, Hanrui Wang, Demi Guo, Song Han

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments Accepted by PMLR

MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models

Linqing Liu, Huan Wang, Jimmy Lin, Richard Socher, Caiming Xiong

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.LG

A Continuous Space Neural Language Model for Bengali Language

Hemayet Ahmed Chowdhury, Md. Azizul Haque Imon, Anisur Rahman, Aisha Khatun, Md. Saiful Islam

专题命中 效率与部署 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments 6 pages