语言大模型 / LLM

Should ChatGPT and Bard Share Revenue with Their Data Providers? A New Business Model for the AI Era

Dong Zhang

专题命中 预训练与数据 ：large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments 22 pages, 8 figures, 2 tables, Published in Advances in Artificial Intelligence and Machine Learning, minor revision made

CMLM-CSE: Based on Conditional MLM Contrastive Learning for Sentence Embeddings

Wei Zhang, Xu Chen

专题命中 预训练与数据 ：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

NoCoLA: The Norwegian Corpus of Linguistic Acceptability

Matias Jentoft, David Samuel

专题命中 预训练与数据 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Published at NoDaLiDa 2023

How Do In-Context Examples Affect Compositional Generalization?

Shengnan An, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Jian-Guang Lou, Dongmei Zhang

专题命中 预训练与数据 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments ACL 2023 main conference, long paper

Absformer: Transformer-based Model for Unsupervised Multi-Document Abstractive Summarization

Mohamed Trabelsi, Huseyin Uzunalioglu

专题命中 预训练与数据 ：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments ICDAR 2023 International Workshop on Machine Learning (WML)

Which Argumentative Aspects of Hate Speech in Social Media can be reliably identified?

Damián Furman, Pablo Torres, José A. Rodríguez, Diego Letzen, Vanina Martínez, Laura Alonso Alemany

专题命中 预训练与数据 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments 9 Pages plus reference and appendix

Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations

Xinxi Lyu, Sewon Min, Iz Beltagy, Luke Zettlemoyer, Hannaneh Hajishirzi

专题命中 预训练与数据 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments 11 pages; 9 figures

Alfred: A System for Prompted Weak Supervision

Peilin Yu, Stephen H. Bach

专题命中 预训练与数据 ：language model(abstract)；prompting(abstract)；分类 cs.CL、cs.LG

Comments ACL 2023 System Demonstration Track

Masked Autoencoding for Scalable and Generalizable Decision Making

Fangchen Liu, Hao Liu, Aditya Grover, Pieter Abbeel

专题命中 预训练与数据 ：language model(abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Accessible Instruction-Following Agent

Kairui Zhou

专题命中 预训练与数据 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

GAP-Gen: Guided Automatic Python Code Generation

Junchen Zhao, Yurun Song, Junlin Wang, Ian G. Harris

专题命中 预训练与数据 ：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop

On the Impossible Safety of Large AI Models

El-Mahdi El-Mhamdi, Sadegh Farhadkhani, Rachid Guerraoui, Nirupam Gupta, Lê-Nguyên Hoang, Rafael Pinot, Sébastien Rouault, John Stephan

专题命中 预训练与数据 ：large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments 40 pages

Three ways to improve feature alignment for open vocabulary detection

Relja Arandjelović, Alex Andonian, Arthur Mensch, Olivier J. Hénaff, Jean-Baptiste Alayrac, Andrew Zisserman

专题命中 预训练与数据 ：language model(abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Salient Span Masking for Temporal Understanding

Jeremy R. Cole, Aditi Chaudhary, Bhuwan Dhingra, Partha Talukdar

专题命中 预训练与数据 ：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments 5 pages 1 figure, to appear in EACL 2023

Linearly Mapping from Image to Text Space

Jack Merullo, Louis Castricato, Carsten Eickhoff, Ellie Pavlick

专题命中 预训练与数据 ：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Accepted at ICLR 2023

Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent

William Merrill, Vivek Ramanujan, Yoav Goldberg, Roy Schwartz, Noah Smith

专题命中 预训练与数据 ：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Appeared at EMNLP 2021. March 7, 2023: Removed irreproducible numbers reported in a footnote with erratum note

The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset

Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic, Margaret Mitchell, Sasha Alexandra Luccioni, Yacine Jernite

专题命中 预训练与数据 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments NeurIPS 2022, Datasets and Benchmarks Track

SantaCoder: don't reach for the stars!

Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, Carolyn Jane Anderson, Yangtian Zi, Joel Lamy Poirier, Hailey Schoelkopf, Sergey Troshin, Dmitry Abulkhanov, Manuel Romero, Michael Lappert, Francesco De Toni, Bernardo García del Río, Qian Liu, Shamik Bose, Urvashi Bhattacharyya, Terry Yue Zhuo, Ian Yu, Paulo Villegas, Marco Zocca, Sourab Mangrulkar, David Lansky, Huu Nguyen, Danish Contractor, Luis Villa, Jia Li, Dzmitry Bahdanau, Yacine Jernite, Sean Hughes, Daniel Fried, Arjun Guha, Harm de Vries, Leandro von Werra

专题命中 预训练与数据 ：large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints

Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, Neil Houlsby

专题命中 预训练与数据 ：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

FewShotTextGCN: K-hop neighborhood regularization for few-shot learning on graphs

Niels van der Heijden, Ekaterina Shutova, Helen Yannakoudakis

专题命中 预训练与数据 ：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments 8 pages, 4 figures, EACL 2023

Compositional generalization in semantic parsing with pretrained transformers

A. Emin Orhan

专题命中 预训练与数据 ：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments v3 adds one reference, adds further discussion, slightly changes formatting

COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning

Yue Yu, Chenyan Xiong, Si Sun, Chao Zhang, Arnold Overwijk

专题命中 预训练与数据 ：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments EMNLP 2022 (Main Conference). The code and Model can be found at https://github.com/OpenMatch/COCO-DR

Journal ref EMNLP 2022

CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning

Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, Steven C. H. Hoi

专题命中 预训练与数据 ：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments An earlier version of the work was accepted to NeurIPS 2022

The Curious Case of Absolute Position Embeddings

Koustuv Sinha, Amirhossein Kazemnejad, Siva Reddy, Joelle Pineau, Dieuwke Hupkes, Adina Williams

专题命中 预训练与数据 ：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Accepted at EMNLP 2022 Findings; 5 pages and 15 pages Appendix

Foundation Transformers

Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, Barun Patra, Zhun Liu, Vishrav Chaudhary, Xia Song, Furu Wei

专题命中 预训练与数据 ：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Work in progress

Matching Text and Audio Embeddings: Exploring Transfer-learning Strategies for Language-based Audio Retrieval

Benno Weck, Miguel Pérez Fernández, Holger Kirchhoff, Xavier Serra

专题命中 预训练与数据 ：foundation model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments 5 pages, 2 figures. Accepted at Detection and Classification of Acoustic Scenes and Events 2022 (DCASE2022)

Enhancing Semantic Understanding with Self-supervised Methods for Abstractive Dialogue Summarization

Hyunjae Lee, Jaewoong Yun, Hyunjin Choi, Seongho Joe, Youngjune L. Gwon

专题命中 预训练与数据 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments 5 pages, 3 figures, INTERSPEECH 2021

Journal ref Proc. Interspeech 2021, 796-800 (2021)

Neural Knowledge Bank for Pretrained Transformers

Damai Dai, Wenbin Jiang, Qingxiu Dong, Yajuan Lyu, Qiaoqiao She, Zhifang Sui

专题命中 预训练与数据 ：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Online Decision Transformer

Qinqing Zheng, Amy Zhang, Aditya Grover

专题命中 预训练与数据 ：language model(abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments Accepted to ICML 2022 (Long Oral Presentation)

RescoreBERT: Discriminative Speech Recognition Rescoring with BERT

Liyan Xu, Yile Gu, Jari Kolehmainen, Haidar Khan, Ankur Gandhe, Ariya Rastrow, Andreas Stolcke, Ivan Bulyko

专题命中 预训练与数据 ：language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Accepted to ICASSP 2022

Journal ref Proc. IEEE ICASSP, May 2022, pp. 6617-6121