语言大模型 / LLM - arXivDaily 专题

2209.15162 2023-03-10 cs.CL cs.LG 73%

Linearly Mapping from Image to Text Space

Jack Merullo, Louis Castricato, Carsten Eickhoff, Ellie Pavlick

专题命中预训练与数据：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Accepted at ICLR 2023

2010.09697 2023-03-09 cs.LG cs.CL 73%

Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent

William Merrill, Vivek Ramanujan, Yoav Goldberg, Roy Schwartz, Noah Smith

专题命中预训练与数据：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Appeared at EMNLP 2021. March 7, 2023: Removed irreproducible numbers reported in a footnote with erratum note

2303.03915 2023-03-08 cs.CL cs.AI 73%

The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset

Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic, Margaret Mitchell, Sasha Alexandra Luccioni, Yacine Jernite

专题命中预训练与数据：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments NeurIPS 2022, Datasets and Benchmarks Track

2301.03988 2023-02-27 cs.SE cs.AI cs.LG 73%

SantaCoder: don't reach for the stars!

Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, Carolyn Jane Anderson, Yangtian Zi, Joel Lamy Poirier, Hailey Schoelkopf, Sergey Troshin, Dmitry Abulkhanov, Manuel Romero, Michael Lappert, Francesco De Toni, Bernardo García del Río, Qian Liu, Shamik Bose, Urvashi Bhattacharyya, Terry Yue Zhuo, Ian Yu, Paulo Villegas, Marco Zocca, Sourab Mangrulkar, David Lansky, Huu Nguyen, Danish Contractor, Luis Villa, Jia Li, Dzmitry Bahdanau, Yacine Jernite, Sean Hughes, Daniel Fried, Arjun Guha, Harm de Vries, Leandro von Werra

专题命中预训练与数据：large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

2212.05055 2023-02-20 cs.LG cs.CL cs.CV 73%

Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints

Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, Neil Houlsby

专题命中预训练与数据：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

2301.10481 2023-02-07 cs.CL cs.LG 73%

FewShotTextGCN: K-hop neighborhood regularization for few-shot learning on graphs

Niels van der Heijden, Ekaterina Shutova, Helen Yannakoudakis

专题命中预训练与数据：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments 8 pages, 4 figures, EACL 2023

2109.15101 2022-12-23 cs.CL cs.LG cs.NE 73%

Compositional generalization in semantic parsing with pretrained transformers

A. Emin Orhan

专题命中预训练与数据：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments v3 adds one reference, adds further discussion, slightly changes formatting

2210.15212 2022-11-28 cs.CL cs.IR cs.LG 73%

COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning

Yue Yu, Chenyan Xiong, Si Sun, Chao Zhang, Arnold Overwijk

专题命中预训练与数据：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments EMNLP 2022 (Main Conference). The code and Model can be found at https://github.com/OpenMatch/COCO-DR

Journal ref EMNLP 2022

2207.01780 2022-11-04 cs.LG cs.CL cs.PL 73%

CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning

Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, Steven C. H. Hoi

专题命中预训练与数据：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments An earlier version of the work was accepted to NeurIPS 2022

2210.12574 2022-10-25 cs.CL cs.LG 73%

The Curious Case of Absolute Position Embeddings

Koustuv Sinha, Amirhossein Kazemnejad, Siva Reddy, Joelle Pineau, Dieuwke Hupkes, Adina Williams

专题命中预训练与数据：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Accepted at EMNLP 2022 Findings; 5 pages and 15 pages Appendix

2210.06423 2022-10-20 cs.LG cs.CL cs.CV 73%

Foundation Transformers

Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, Barun Patra, Zhun Liu, Vishrav Chaudhary, Xia Song, Furu Wei

专题命中预训练与数据：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Work in progress

2210.02833 2022-10-07 cs.IR cs.CL cs.LG cs.SD eess.AS 73%

Matching Text and Audio Embeddings: Exploring Transfer-learning Strategies for Language-based Audio Retrieval

Benno Weck, Miguel Pérez Fernández, Holger Kirchhoff, Xavier Serra

专题命中预训练与数据：foundation model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments 5 pages, 2 figures. Accepted at Detection and Classification of Acoustic Scenes and Events 2022 (DCASE2022)

2209.00278 2022-09-02 cs.CL cs.AI 73%

Enhancing Semantic Understanding with Self-supervised Methods for Abstractive Dialogue Summarization

Hyunjae Lee, Jaewoong Yun, Hyunjin Choi, Seongho Joe, Youngjune L. Gwon

专题命中预训练与数据：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments 5 pages, 3 figures, INTERSPEECH 2021

Journal ref Proc. Interspeech 2021, 796-800 (2021)

2208.00399 2022-08-17 cs.CL cs.AI 73%

Neural Knowledge Bank for Pretrained Transformers

Damai Dai, Wenbin Jiang, Qingxiu Dong, Yajuan Lyu, Qiaoqiao She, Zhifang Sui

专题命中预训练与数据：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

2202.05607 2022-07-14 cs.LG cs.AI 73%

Online Decision Transformer

Qinqing Zheng, Amy Zhang, Aditya Grover

专题命中预训练与数据：language model(abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments Accepted to ICML 2022 (Long Oral Presentation)

2202.01094 2022-07-13 eess.AS cs.CL cs.LG cs.SD 73%

RescoreBERT: Discriminative Speech Recognition Rescoring with BERT

Liyan Xu, Yile Gu, Jari Kolehmainen, Haidar Khan, Ankur Gandhe, Ariya Rastrow, Andreas Stolcke, Ivan Bulyko

专题命中预训练与数据：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Accepted to ICASSP 2022

Journal ref Proc. IEEE ICASSP, May 2022, pp. 6617-6121

2206.11953 2022-06-27 cs.CL cs.AI 73%

Do Trajectories Encode Verb Meaning?

Dylan Ebert, Chen Sun, Ellie Pavlick

专题命中预训练与数据：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments NAACL 2022

2206.05802 2022-06-15 cs.CL cs.LG 73%

Self-critiquing models for assisting human evaluators

William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, Jan Leike

专题命中预训练与数据：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

2206.04591 2022-06-10 cs.CL cs.CR cs.LG 73%

Privacy Leakage in Text Classification: A Data Extraction Approach

Adel Elmahdy, Huseyin A. Inan, Robert Sim

专题命中预训练与数据：language model(abstract)；post-training(abstract)；分类 cs.CL、cs.LG

Comments 8 pages, 4 tables. Accepted at NAACL 2022 Workshop on Privacy in NLP (PrivateNLP)

2205.10228 2022-05-23 cs.CL cs.CR cs.LG 73%

You Don't Know My Favorite Color: Preventing Dialogue Representations from Revealing Speakers' Private Personas

Haoran Li, Yangqiu Song, Lixin Fan

专题命中预训练与数据：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Conference paper accepted by NAACL 2022

2205.05543 2022-05-12 cs.CV cs.AI cs.LG 73%

An Empirical Study Of Self-supervised Learning Approaches For Object Detection With Transformers

Gokul Karthik Kumar, Sahal Shaji Mullappilly, Abhishek Singh Gehlot

专题命中预训练与数据：language model(abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments Final Project for the course "Visual Object Detection And Recognition" (CV703) at MBZUAI

2205.04086 2022-05-10 cs.CL cs.AI 73%

A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bank

Dan Malkin, Tomasz Limisiewicz, Gabriel Stanovsky

专题命中预训练与数据：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments Accepted to NAACL 2022

2205.03873 2022-05-10 cs.CV cs.AI cs.LG 73%

Multimodal Semi-Supervised Learning for Text Recognition

Aviad Aberdam, Roy Ganz, Shai Mazor, Ron Litman

专题命中预训练与数据：language model(abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments Code will be published upon publication

2109.04223 2022-05-06 cs.CL cs.AI 73%

KELM: Knowledge Enhanced Pre-Trained Language Representations with Message Passing on Hierarchical Relational Graphs

Yinquan Lu, Haonan Lu, Guirong Fu, Qun Liu

专题命中预训练与数据：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments ICLR 2022 on DLG4NLP

Journal ref ICLR 2022 Workshop on Deep Learning on Graphs for Natural Language Processing,2022

2203.17189 2022-04-01 cs.LG cs.CL 73%

Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$

Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, Andrea Gesmundo

专题命中预训练与数据：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

2203.08242 2022-03-17 cs.CL cs.LG 73%

Data Contamination: From Memorization to Exploitation

Inbal Magar, Roy Schwartz

专题命中预训练与数据：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Accepted to ACL 2022

2203.07785 2022-03-16 cs.CL cs.AI 73%

The Ghost in the Machine has an American accent: value conflict in GPT-3

Rebecca L Johnson, Giada Pistilli, Natalia Menédez-González, Leslye Denisse Dias Duran, Enrico Panai, Julija Kalpokiene, Donald Jay Bertulfo

专题命中预训练与数据：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments There are a total of 15 pages of the PDF including 8 pages of the main manuscript, 3 pages of references, and 4 pages of appendices. The paper is currently under review by a conference

2107.07653 2022-03-15 cs.CL cs.AI 73%

TAPEX: Table Pre-training via Learning a Neural SQL Executor

Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, Jian-Guang Lou

专题命中预训练与数据：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments ICLR 2022 camera ready version

2203.03131 2022-03-08 cs.CL cs.AI 73%

Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models

Shengnan An, Yifei Li, Zeqi Lin, Qian Liu, Bei Chen, Qiang Fu, Weizhu Chen, Nanning Zheng, Jian-Guang Lou

专题命中预训练与数据：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

2109.07437 2022-02-08 cs.LG cs.CL 73%

Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative

Lucio M. Dery, Paul Michel, Ameet Talwalkar, Graham Neubig

专题命中预训练与数据：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments 18 pages, 4 figures

1. 预训练与数据 12486 篇

Linearly Mapping from Image to Text Space

Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent

The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset

SantaCoder: don't reach for the stars!

Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints

FewShotTextGCN: K-hop neighborhood regularization for few-shot learning on graphs

Compositional generalization in semantic parsing with pretrained transformers

COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning

CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning

The Curious Case of Absolute Position Embeddings

Foundation Transformers

Matching Text and Audio Embeddings: Exploring Transfer-learning Strategies for Language-based Audio Retrieval

Enhancing Semantic Understanding with Self-supervised Methods for Abstractive Dialogue Summarization

Neural Knowledge Bank for Pretrained Transformers

Online Decision Transformer

RescoreBERT: Discriminative Speech Recognition Rescoring with BERT

Do Trajectories Encode Verb Meaning?

Self-critiquing models for assisting human evaluators

Privacy Leakage in Text Classification: A Data Extraction Approach

You Don't Know My Favorite Color: Preventing Dialogue Representations from Revealing Speakers' Private Personas

An Empirical Study Of Self-supervised Learning Approaches For Object Detection With Transformers

A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bank

Multimodal Semi-Supervised Learning for Text Recognition

KELM: Knowledge Enhanced Pre-Trained Language Representations with Message Passing on Hierarchical Relational Graphs

Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$

Data Contamination: From Memorization to Exploitation

The Ghost in the Machine has an American accent: value conflict in GPT-3

TAPEX: Table Pre-training via Learning a Neural SQL Executor

Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models

Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative