语言大模型 / LLM - arXivDaily 专题

2204.12052 2022-04-27 cs.CL 83%

Pretraining Chinese BERT for Detecting Word Insertion and Deletion Errors

Cong Zhou, Yong Dai, Duyu Tang, Enbo Zhao, Zhangyin Feng, Li Kuang, Shuming Shi

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

Comments 12 pages

2109.12573 2022-04-26 cs.CL 83%

XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge

Xiaoze Jiang, Yaobo Liang, Weizhu Chen, Nan Duan

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments AAAI-2022

2106.15110 2022-04-26 cs.CL 83%

Time-Aware Language Models as Temporal Knowledge Bases

Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, William W. Cohen

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments Version accepted to TACL

Journal ref Transactions of the Association for Computational Linguistics 2022; 10 257-273

2107.14600 2022-04-19 cs.CL 83%

MDQE: A More Accurate Direct Pretraining for Machine Translation Quality Estimation

Lei Lin

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

Comments Just some ideas of my own, not supported by experiments

2204.04504 2022-04-12 cs.CL 83%

TANet: Thread-Aware Pretraining for Abstractive Conversational Summarization

Ze Yang, Liran Wang, Zhoujin Tian, Wei Wu, Zhoujun Li

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

Comments NAACL2022-findings

2201.02026 2022-04-06 cs.CL 83%

Fortunately, Discourse Markers Can Enhance Language Models for Sentiment Analysis

Liat Ein-Dor, Ilya Shnayderman, Artem Spector, Lena Dankin, Ranit Aharonov, Noam Slonim

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments Published in AAAI 2022

2110.08151 2022-03-31 cs.CL 83%

mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models

Ryokan Ri, Ikuya Yamada, Yoshimasa Tsuruoka

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments ACL 2022

2202.12024 2022-03-24 cs.CL 83%

NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Better

Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang, Xing Xie

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments ACL 2022

2103.01819 2022-01-04 cs.CL 83%

The Rediscovery Hypothesis: Language Models Need to Meet Linguistics

Vassilina Nikoulina, Maxat Tezekbayev, Nuradil Kozhakhmet, Madina Babazhanova, Matthias Gallé, Zhenisbek Assylbekov

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Journal ref Journal of Artificial Intelligence Vol. 72 (2021) 1343-1384

2107.00676 2021-12-24 cs.CL 83%

A Primer on Pretrained Multilingual Language Models

Sumanth Doddapaneni, Gowtham Ramesh, Mitesh M. Khapra, Anoop Kunchukuttan, Pratyush Kumar

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

2112.07254 2021-12-15 eess.AS cs.CL cs.SD 83%

Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Model

Keqi Deng, Songjun Cao, Yike Zhang, Long Ma

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments ASRU2021

2112.02839 2021-12-07 cs.MM cs.AI 83%

MoCA: Incorporating Multi-stage Domain Pretraining and Cross-guided Multimodal Attention for Textbook Question Answering

Fangzhi Xu, Qika Lin, Jun Liu, Lingling Zhang, Tianzhe Zhao, Qi Chai, Yudai Pan

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.AI

Comments 9 pages, 6 figures

2111.06053 2021-11-12 cs.CL 83%

Improving Large-scale Language Models and Resources for Filipino

Jan Christian Blaise Cruz, Charibeth Cheng

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments Resources are available at blaisecruz.com/resources

2109.04513 2021-10-26 cs.CL 83%

Filling the Gaps in Ancient Akkadian Texts: A Masked Language Modelling Approach

Koren Lazar, Benny Saret, Asaf Yehudai, Wayne Horowitz, Nathan Wasserman, Gabriel Stanovsky

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments Accepted to EMNLP 2021 (Main Conference)

2110.08182 2021-10-18 cs.CL cs.CV 83%

The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Color

Cory Paik, Stéphane Aroca-Ouellette, Alessandro Roncone, Katharina Kann

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments Accepted to EMNLP 2021, 9 Pages

2110.07143 2021-10-15 cs.CL 83%

bert2BERT: Towards Reusable Pretrained Language Models

Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, Qun Liu

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL

2110.05301 2021-10-12 cs.CL 83%

On a Benefit of Mask Language Modeling: Robustness to Simplicity Bias

Ting-Rui Chiang

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments Work in progress

2007.08426 2021-09-28 cs.CL 83%

Investigating Pretrained Language Models for Graph-to-Text Generation

Leonardo F. R. Ribeiro, Martin Schmitt, Hinrich Schütze, Iryna Gurevych

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments Accepted as a long paper to NLP4ConvAI, EMNLP2021

2109.07301 2021-09-16 cs.CL 83%

What Vision-Language Models `See' when they See Scenes

Michele Cafagna, Kees van Deemter, Albert Gatt

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

2109.06605 2021-09-15 cs.CL 83%

MDAPT: Multilingual Domain Adaptive Pretraining in a Single Model

Rasmus Kær Jørgensen, Mareike Hartmann, Xiang Dai, Desmond Elliott

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

Comments Findings of EMNLP 2021

2109.00055 2021-09-14 cs.CL 83%

Sentence Bottleneck Autoencoders from Transformer Language Models

Ivan Montero, Nikolaos Pappas, Noah A. Smith

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

2012.15562 2021-09-13 cs.CL 83%

UNKs Everywhere: Adapting Multilingual Language Models to New Scripts

Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, Sebastian Ruder

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments EMNLP 2021

2109.03646 2021-09-09 cs.CL 83%

Sustainable Modular Debiasing of Language Models

Anne Lauscher, Tobias Lüken, Goran Glavaš

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments Accepted for EMNLP-Findings 2021

2109.03630 2021-09-09 cs.CL 83%

Discrete and Soft Prompting for Multilingual Models

Mengjie Zhao, Hinrich Schütze

专题命中预训练与数据：prompting(title,abstract)；language model(abstract)；分类 cs.CL

Comments EMNLP 2021

2101.00406 2021-09-06 cs.CL 83%

CDLM: Cross-Document Language Modeling

Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E. Peters, Arie Cattan, Ido Dagan

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments EMNLP 2021, findings

2108.12226 2021-08-30 cs.CL cs.SD eess.AS 83%

Injecting Text in Self-Supervised Speech Pretraining

Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Gary Wang, Pedro Moreno

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

Comments submit to ASRU 2021

2106.05634 2021-06-11 cs.CL 83%

Exploring Unsupervised Pretraining Objectives for Machine Translation

Christos Baziotis, Ivan Titov, Alexandra Birch, Barry Haddow

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

Comments Findings of ACL 2021

2012.15613 2021-06-03 cs.CL 83%

How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models

Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, Iryna Gurevych

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments ACL 2021

2012.14763 2021-06-01 cs.CL 83%

CMV-BERT: Contrastive multi-vocab pretraining of BERT

Wei Zhu, Daniel Cheung

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL

Comments will add more detailed technical contents, and more detailed experiments

2105.00572 2021-05-04 cs.CL 83%

Larger-Scale Transformers for Multilingual Masked Language Modeling

Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, Alexis Conneau

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments 4 pages

1. 预训练与数据 12403 篇

Pretraining Chinese BERT for Detecting Word Insertion and Deletion Errors

XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge

Time-Aware Language Models as Temporal Knowledge Bases

MDQE: A More Accurate Direct Pretraining for Machine Translation Quality Estimation

TANet: Thread-Aware Pretraining for Abstractive Conversational Summarization

Fortunately, Discourse Markers Can Enhance Language Models for Sentiment Analysis

mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models

NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Better

The Rediscovery Hypothesis: Language Models Need to Meet Linguistics

A Primer on Pretrained Multilingual Language Models

Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Model

MoCA: Incorporating Multi-stage Domain Pretraining and Cross-guided Multimodal Attention for Textbook Question Answering

Improving Large-scale Language Models and Resources for Filipino

Filling the Gaps in Ancient Akkadian Texts: A Masked Language Modelling Approach

The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Color

bert2BERT: Towards Reusable Pretrained Language Models

On a Benefit of Mask Language Modeling: Robustness to Simplicity Bias

Investigating Pretrained Language Models for Graph-to-Text Generation

What Vision-Language Models `See' when they See Scenes

MDAPT: Multilingual Domain Adaptive Pretraining in a Single Model

Sentence Bottleneck Autoencoders from Transformer Language Models

UNKs Everywhere: Adapting Multilingual Language Models to New Scripts

Sustainable Modular Debiasing of Language Models

Discrete and Soft Prompting for Multilingual Models

CDLM: Cross-Document Language Modeling

Injecting Text in Self-Supervised Speech Pretraining

Exploring Unsupervised Pretraining Objectives for Machine Translation

How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models

CMV-BERT: Contrastive multi-vocab pretraining of BERT

Larger-Scale Transformers for Multilingual Masked Language Modeling