语言大模型 / LLM

Safe-CLIP: Removing NSFW Concepts from Vision-and-Language Models

Samuele Poppi, Tobia Poppi, Federico Cocchi, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments ECCV 2024

The Silent Curriculum: How Does LLM Monoculture Shape Educational Content and Its Accessibility?

Aman Priyanshu, Supriti Vijay

专题命中 预训练与数据 ：LLM(title)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments 5 pages and 4 figures. Accepted at The Workshop on Global AI Cultures at the International Conference on Learning Representations, 2024 (ICLR'24)

Super Tiny Language Models

Dylan Hillier, Leon Guertler, Cheston Tan, Palaash Agrawal, Chen Ruirui, Bobby Cheng

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments 11 pages, 4 figures

DE-COP: Detecting Copyrighted Content in Language Models Training Data

André V. Duarte, Xuandong Zhao, Arlindo L. Oliveira, Lei Li

专题命中 预训练与数据 ：language model(title,abstract)；LLM(abstract)；分类 cs.CL、cs.LG

Exploring Spatial Representations in the Historical Lake District Texts with LLM-based Relation Extraction

Erum Haris, Anthony G. Cohn, John G. Stell

专题命中 预训练与数据 ：LLM(title)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Knowledge Fusion By Evolving Weights of Language Models

Guodong Du, Jing Li, Hanting Liu, Runhua Jiang, Shuyang Yu, Yifei Guo, Sim Kuan Goh, Ho-Kin Tang

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted by ACL2024 Findings

Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale

Xiang Hu, Pengyu Ji, Qingyang Zhu, Wei Wu, Kewei Tu

专题命中 预训练与数据 ：language model(title,abstract)；SLM(abstract)；分类 cs.CL、cs.AI

Comments accepted by ACL 2024

Unlearning Traces the Influential Training Data of Language Models

Masaru Isonuma, Ivan Titov

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments 14 pages, to appear in ACL2024 main conference (long paper)

In-Context Unlearning: Language Models as Few Shot Unlearners

Martin Pawelczyk, Seth Neel, Himabindu Lakkaraju

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.AI、cs.LG

Comments Accepted at ICML 2024

Balanced Data Sampling for Language Model Training with Clustering

Yunfan Shao, Linyang Li, Zhaoye Fei, Hang Yan, Dahua Lin, Xipeng Qiu

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments ACL 2024 (findings), Code is released at https://github.com/choosewhatulike/cluster-clip

DocReLM: Mastering Document Retrieval with Language Model

Gengchen Wei, Xinle Pang, Tianning Zhang, Yu Sun, Xun Qian, Chen Lin, Han-Sen Zhong, Wanli Ouyang

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

LMD3: Language Model Data Density Dependence

John Kirchenbauer, Garrett Honke, Gowthami Somepalli, Jonas Geiping, Daphne Ippolito, Katherine Lee, Tom Goldstein, David Andre

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments 10 pages in the main body

Türkçe Dil Modellerinin Performans Karşılaştırması Performance Comparison of Turkish Language Models

Eren Dogan, M. Egemen Uzun, Atahan Uz, H. Emre Seyrek, Ahmed Zeer, Ezgi Sevi, H. Toprak Kesgin, M. Kaan Yuce, M. Fatih Amasyali

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments in Turkish language. Bazı çalışmaları içermediğini söyleyen hakem yorumu nedeniyle bir konferanstan kabul almadı. Ancak hakemin bahsettiği çalışmalar bildiri gönderme son tarihinde yayınlanmamıştı

Language Models Implement Simple Word2Vec-style Vector Arithmetic

Jack Merullo, Carsten Eickhoff, Ellie Pavlick

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments NAACL

Emergent Abilities in Reduced-Scale Generative Language Models

Sherin Muckatira, Vijeta Deshpande, Vladislav Lialin, Anna Rumshisky

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

Comments 16 pages, 4 figures. Accepted to NAACL 2024 Findings

Bridging Remote Sensors with Multisensor Geospatial Foundation Models

Boran Han, Shuai Zhang, Xingjian Shi, Markus Reichstein

专题命中 预训练与数据 ：foundation model(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments Accepted to CVPR

Sequence-to-Sequence Language Models for Character and Emotion Detection in Dream Narratives

Gustave Cortal

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

A Knowledge-Injected Curriculum Pretraining Framework for Question Answering

Xin Lin, Tianhuang Su, Zhenya Huang, Shangzi Xue, Haifeng Liu, Enhong Chen

专题命中 预训练与数据 ：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted by WWW 2024

On the Scaling Laws of Geographical Representation in Language Models

Nathan Godey, Éric de la Clergerie, Benoît Sagot

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted at LREC-COLING 2024

Inconsistencies in Masked Language Models

Tom Young, Yunan Chen, Yang You

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Embracing the black box: Heading towards foundation models for causal discovery from time series data

Gideon Stein, Maha Shadaydeh, Joachim Denzler

专题命中 预训练与数据 ：foundation model(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments AAAI Workshop (AI4TS) 2024

Dynamic Masking Rate Schedules for MLM Pretraining

Zachary Ankner, Naomi Saphra, Davis Blalock, Jonathan Frankle, Matthew L. Leavitt

专题命中 预训练与数据 ：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI

The effectiveness of MAE pre-pretraining for billion-scale pretraining

Mannat Singh, Quentin Duval, Kalyan Vasudev Alwala, Haoqi Fan, Vaibhav Aggarwal, Aaron Adcock, Armand Joulin, Piotr Dollár, Christoph Feichtenhofer, Ross Girshick, Rohit Girdhar, Ishan Misra

专题命中 预训练与数据 ：pretraining(title,abstract)；foundation model(abstract)；分类 cs.AI、cs.LG

Comments ICCV 2023. Models available at https://github.com/facebookresearch/maws/

EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation Model

Yuqi Chen, Kan Ren, Kaitao Song, Yansen Wang, Yifan Wang, Dongsheng Li, Lili Qiu

专题命中 预训练与数据 ：foundation model(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments A preprint version of an ongoing work

MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining

Jacob Portes, Alex Trott, Sam Havens, Daniel King, Abhinav Venigalla, Moin Nadeem, Nikhil Sardana, Daya Khudia, Jonathan Frankle

专题命中 预训练与数据 ：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments 10 pages, 4 figures in main text. 25 pages total

Journal ref NeurIPS 2023

Improving Language Plasticity via Pretraining with Active Forgetting

Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Ifeoluwa Adelani, Pontus Stenetorp, Sebastian Riedel, Mikel Artetxe

专题命中 预训练与数据 ：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments NeurIPS 2023 Final Version

Data Portraits: Recording Foundation Model Training Data

Marc Marone, Benjamin Van Durme

专题命中 预训练与数据 ：foundation model(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments NeurIPS 2023 Datasets and Benchmarks

Data Selection for Language Models via Importance Resampling

Sang Michael Xie, Shibani Santurkar, Tengyu Ma, Percy Liang

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments NeurIPS 2023

Instructed Language Models with Retrievers Are Powerful Entity Linkers

Zilin Xiao, Ming Gong, Jie Wu, Xingyao Zhang, Linjun Shou, Jian Pei, Daxin Jiang

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted to EMNLP 2023 Main

FloodBrain: Flood Disaster Reporting by Web-based Retrieval Augmented Generation with an LLM

Grace Colverd, Paul Darm, Leonard Silverberg, Noah Kasmanoff

专题命中 预训练与数据 ：LLM(title)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Version is the one submitted to Artificial Intelligence for Humanitarian Assistance and Disaster Response Workshop @Neurips2023. All authors contributed equally to this work