语言大模型 / LLM - arXivDaily 专题

2211.17201 2022-12-01 cs.CL cs.LG math.OC 84%

ExtremeBERT: A Toolkit for Accelerating Pretraining of Customized BERT

Rui Pan, Shizhe Diao, Jianlin Chen, Tong Zhang

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

2112.10668 2022-11-11 cs.CL cs.AI 84%

Few-shot Learning with Multilingual Language Models

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li

专题命中预训练与数据：language model(title,abstract)；prompting(abstract)；分类 cs.CL、cs.AI

Comments Accepted to EMNLP 2022; 34 pages

2201.06910 2022-11-01 cs.LG cs.CL 84%

ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalization

Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang, Haiyu Li, Zhilin Yang

专题命中预训练与数据：pretraining(title,abstract)；prompting(abstract)；分类 cs.CL、cs.LG

Comments 18 pages

2111.10962 2022-10-20 cs.CL cs.AI 84%

Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triples

Linlin Liu, Xin Li, Ruidan He, Lidong Bing, Shafiq Joty, Luo Si

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments Accepted by EMNLP 2022

2208.02812 2022-10-13 cs.CV cs.AI cs.LG 84%

P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting

Ziyi Wang, Xumin Yu, Yongming Rao, Jie Zhou, Jiwen Lu

专题命中预训练与数据：prompting(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments Accepted to NeurIPS 2022, project page: https://p2p.ivg-research.xyz

2202.04538 2022-10-13 cs.CL cs.LG 84%

Generating Training Data with Language Models: Towards Zero-Shot Language Understanding

Yu Meng, Jiaxin Huang, Yu Zhang, Jiawei Han

专题命中预训练与数据：language model(title,abstract)；prompting(abstract)；分类 cs.CL、cs.LG

Comments NeurIPS 2022. (Code: https://github.com/yumeng5/SuperGen)

2209.01712 2022-09-07 cs.LG cs.AI q-bio.BM 84%

ChemBERTa-2: Towards Chemical Foundation Models

Walid Ahmad, Elana Simon, Seyone Chithrananda, Gabriel Grand, Bharath Ramsundar

专题命中预训练与数据：foundation model(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments ELLIS Machine Learning for Molecule Discovery Workshop

2111.04130 2022-07-25 cs.CL cs.LG 84%

NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework

Xingcheng Yao, Yanan Zheng, Xiaocong Yang, Zhilin Yang

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments 14 pages, 5 figures

Journal ref Proceedings of the 39th International Conference on Machine Learning, PMLR 162:25438-25451, 2022

2207.10245 2022-07-22 cs.CL cs.AI 84%

The Birth of Bias: A case study on the evolution of gender bias in an English language model

Oskar van der Wal, Jaap Jumelet, Katrin Schulz, Willem Zuidema

专题命中预训练与数据：language model(title,abstract)；small language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted at the 4th Workshop on Gender Bias in Natural Language Processing (NAACL, 2022)

2206.14355 2022-06-30 cs.CV cs.CL cs.LG 84%

EBMs vs. CL: Exploring Self-Supervised Visual Pretraining for Visual Question Answering

Violetta Shevchenko, Ehsan Abbasnejad, Anthony Dick, Anton van den Hengel, Damien Teney

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

2205.01863 2022-06-27 cs.CL cs.CR cs.LG 84%

Provably Confidential Language Modelling

Xuandong Zhao, Lei Li, Yu-Xiang Wang

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

Comments NAACL 2022

2204.05939 2022-04-13 cs.CL cs.AI 84%

Mining Logical Event Schemas From Pre-Trained Language Models

Lane Lawley, Lenhart Schubert

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments To appear at ACL SRW 2022

2111.04909 2022-04-12 cs.CL cs.AI 84%

FPM: A Collection of Large-scale Foundation Pre-trained Language Models

Dezhou Shen

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments 12 pages, 4 tables

2204.03243 2022-04-08 cs.CL cs.LG 84%

Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators

Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, Xia Song

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments ICLR 2022. (Code and Models: https://github.com/microsoft/AMOS)

2203.13240 2022-03-25 cs.CL cs.LG 84%

Token Dropping for Efficient BERT Pretraining

Le Hou, Richard Yuanzhe Pang, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, Denny Zhou

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments ACL 2022

2110.04541 2022-03-22 cs.CL cs.LG 84%

The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design

Yoav Levine, Noam Wies, Daniel Jannai, Dan Navon, Yedid Hoshen, Amnon Shashua

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

2110.09456 2021-11-02 cs.CL cs.AI 84%

NormFormer: Improved Transformer Pretraining with Extra Normalization

Sam Shleifer, Jason Weston, Myle Ott

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI

2102.04130 2021-10-29 cs.CL cs.AI 84%

Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models

Hannah Kirk, Yennie Jun, Haider Iqbal, Elias Benussi, Filippo Volpin, Frederic A. Dreyer, Aleksandar Shtedritski, Yuki M. Asano

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted to NeurIPS 2021. Code and data at https://github.com/oxai/intersectional_gpt2

2110.10319 2021-10-22 cs.CL cs.CY cs.IR cs.LG 84%

LMSOC: An Approach for Socially Sensitive Pretraining

Vivek Kulkarni, Shubhanshu Mishra, Aria Haghighi

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Camera ready version. Accepted to EMNLP 2021 Findings. Code for reproducing the experiments can be found at: https://github.com/twitter-research/lmsoc

2104.07540 2021-10-05 cs.CL cs.LG 84%

Generating Datasets with Pretrained Language Models

Timo Schick, Hinrich Schütze

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Accepted at EMNLP2021

2109.00895 2021-09-03 cs.CV cs.AI cs.CL 84%

Knowledge Perceived Multi-modal Pretraining in E-commerce

Yushan Zhu, Huaixiao Tou, Wen Zhang, Ganqiang Ye, Hui Chen, Ningyu Zhang, Huajun Chen

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted to ACM MM 2021

2108.02170 2021-08-05 cs.CL cs.AI 84%

Curriculum learning for language modeling

Daniel Campos

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

2107.10474 2021-07-23 cs.CL cs.LG 84%

Back-Translated Task Adaptive Pretraining: Improving Accuracy and Robustness on Text Classification

Junghoon Lee, Jounghee Kim, Pilsung Kang

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

2105.13665 2021-05-31 cs.CL cs.AI 84%

Domain-Adaptive Pretraining Methods for Dialogue Understanding

Han Wu, Kun Xu, Linfeng Song, Lifeng Jin, Haisong Zhang, Linqi Song

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments 6 pages, to appear in ACL2021

2105.01735 2021-05-06 cs.CL cs.LG 84%

HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish

Robert Mroczkowski, Piotr Rybak, Alina Wróblewska, Ireneusz Gawlik

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Published in Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing

2102.12982 2021-02-26 cs.CL cs.AI cs.CV 84%

A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectives

Nils Rethmeier, Isabelle Augenstein

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI

2010.15980 2020-11-10 cs.CL cs.LG 84%

AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts

Taylor Shin, Yasaman Razeghi, Robert L. Logan, Eric Wallace, Sameer Singh

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments v2: Fixed error in Figure 2

2006.08671 2020-06-17 cs.CL cs.LG stat.ML 84%

To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Tasks

Sinong Wang, Madian Khabsa, Hao Ma

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Accepted in ACL2020

2005.09207 2020-05-28 cs.IR cs.CL cs.LG 84%

Table Search Using a Deep Contextualized Language Model

Zhiyu Chen, Mohamed Trabelsi, Jeff Heflin, Yinan Xu, Brian D. Davison

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Accepted at SIGIR 2020 (Long)

2005.00318 2020-05-04 cs.CL cs.LG 84%

Can Multilingual Language Models Transfer to an Unseen Dialect? A Case Study on North African Arabizi

Benjamin Muller, Benoit Sagot, Djamé Seddah

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

1. 预训练与数据 12429 篇

ExtremeBERT: A Toolkit for Accelerating Pretraining of Customized BERT

Few-shot Learning with Multilingual Language Models

ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalization

Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triples

P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting

Generating Training Data with Language Models: Towards Zero-Shot Language Understanding

ChemBERTa-2: Towards Chemical Foundation Models

NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework

The Birth of Bias: A case study on the evolution of gender bias in an English language model

EBMs vs. CL: Exploring Self-Supervised Visual Pretraining for Visual Question Answering

Provably Confidential Language Modelling

Mining Logical Event Schemas From Pre-Trained Language Models

FPM: A Collection of Large-scale Foundation Pre-trained Language Models

Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators

Token Dropping for Efficient BERT Pretraining

The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design

NormFormer: Improved Transformer Pretraining with Extra Normalization

Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models

LMSOC: An Approach for Socially Sensitive Pretraining

Generating Datasets with Pretrained Language Models

Knowledge Perceived Multi-modal Pretraining in E-commerce

Curriculum learning for language modeling

Back-Translated Task Adaptive Pretraining: Improving Accuracy and Robustness on Text Classification

Domain-Adaptive Pretraining Methods for Dialogue Understanding

HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish

A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectives

AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts

To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Tasks

Table Search Using a Deep Contextualized Language Model

Can Multilingual Language Models Transfer to an Unseen Dialect? A Case Study on North African Arabizi