语言大模型 / LLM - arXivDaily 专题

2306.11702 2023-09-04 cs.DB cs.CL 88%

Lingua Manga: A Generic Large Language Model Centric System for Data Curation

Zui Chen, Lei Cao, Sam Madden

专题命中预训练与数据：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL

Comments 4 pages, 6 figures, VLDB 2023 Demo paper

2306.03950 2023-07-10 cs.CL 88%

MISGENDERED: Limits of Large Language Models in Understanding Pronouns

Tamanna Hossain, Sunipa Dev, Sameer Singh

专题命中预训练与数据：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL

Comments Accepted at ACL 2023 as a long paper

2305.08283 2023-07-07 cs.CL 88%

From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models

Shangbin Feng, Chan Young Park, Yuhan Liu, Yulia Tsvetkov

专题命中预训练与数据：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL

Comments ACL 2023

2306.11892 2023-06-22 cs.CL 88%

Exploring New Frontiers in Agricultural NLP: Investigating the Potential of Large Language Models for Food Applications

Saed Rezayi, Zhengliang Liu, Zihao Wu, Chandra Dhakal, Bao Ge, Haixing Dai, Gengchen Mai, Ninghao Liu, Chen Zhen, Tianming Liu, Sheng Li

专题命中预训练与数据：language model(title,abstract)；large language model(title)；pretraining(abstract)；分类 cs.CL

2205.06733 2023-06-12 cs.CL 88%

Arithmetic-Based Pretraining -- Improving Numeracy of Pretrained Language Models

Dominic Petrak, Nafise Sadat Moosavi, Iryna Gurevych

专题命中预训练与数据：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL

Comments Published at StarSEM2023

2305.10036 2023-06-05 cs.CL cs.CY 88%

Are You Copying My Model? Protecting the Copyright of Large Language Models for EaaS via Backdoor Watermark

Wenjun Peng, Jingwei Yi, Fangzhao Wu, Shangxi Wu, Bin Zhu, Lingjuan Lyu, Binxing Jiao, Tong Xu, Guangzhong Sun, Xing Xie

专题命中预训练与数据：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL

Comments Accepted by ACL 2023

2304.11158 2023-06-02 cs.CL 88%

Emergent and Predictable Memorization in Large Language Models

Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, Edward Raff

专题命中预训练与数据：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL

2304.01373 2023-06-01 cs.CL 88%

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, Oskar van der Wal

专题命中预训练与数据：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL

Comments Code at https://github.com/EleutherAI/pythia

2305.15275 2023-05-25 cs.CL 88%

Self-Evolution Learning for Discriminative Language Model Pretraining

Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, Dacheng Tao

专题命中预训练与数据：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL

Comments Accepted to Findings of ACL2023

2305.15041 2023-05-25 cs.CL 88%

Generating Faithful Synthetic Data with Large Language Models: A Case Study in Computational Social Science

Veniamin Veselovsky, Manoel Horta Ribeiro, Akhil Arora, Martin Josifoski, Ashton Anderson, Robert West

专题命中预训练与数据：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL

Comments 8 pages

2305.12816 2023-05-23 cs.CL 88%

Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model

Xiao Wang, Weikang Zhou, Qi Zhang, Jie Zhou, Songyang Gao, Junzhe Wang, Menghan Zhang, Xiang Gao, Yunwen Chen, Tao Gui

专题命中预训练与数据：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL

Comments Accepted by ACL2023

2305.02156 2023-05-04 cs.IR cs.CL 88%

Zero-Shot Listwise Document Reranking with a Large Language Model

Xueguang Ma, Xinyu Zhang, Ronak Pradeep, Jimmy Lin

专题命中预训练与数据：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL

2304.10447 2023-04-21 cs.CL 88%

Domain-specific Continued Pretraining of Language Models for Capturing Long Context in Mental Health

Shaoxiong Ji, Tianlin Zhang, Kailai Yang, Sophia Ananiadou, Erik Cambria, Jörg Tiedemann

专题命中预训练与数据：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL

2205.10770 2022-11-04 cs.CL 88%

Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models

Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, Armen Aghajanyan

专题命中预训练与数据：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL

2112.02125 2022-08-16 cs.CR cs.AI 88%

Examining Zero-Shot Vulnerability Repair with Large Language Models

Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, Brendan Dolan-Gavitt

专题命中预训练与数据：large language model(title,abstract)；language model(title,abstract)；分类 cs.AI

Comments 18 pages, 19 figures. Accepted for publication in 2023 IEEE Symposium on Security and Privacy (SP)

2110.08534 2022-07-20 cs.CL 88%

Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora

Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, Xiang Ren

专题命中预训练与数据：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL

Comments Accepted at NAACL 2022; fixed Figure 7 (a)(b) in Appendix

2204.13509 2022-05-10 cs.CL 88%

On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model

Seongjin Shin, Sang-Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung-Woo Ha, Nako Sung

专题命中预训练与数据：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL

Comments Accepted to NAACL2022 as a long paper. Camera-ready version

2112.08709 2022-05-06 cs.CL 88%

DOCmT5: Document-Level Pretraining of Multilingual Language Models

Chia-Hsuan Lee, Aditya Siddhant, Viresh Ratnakar, Melvin Johnson

专题命中预训练与数据：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL

Comments NAACL 2022 Findings

2203.10326 2022-03-23 cs.CL 88%

Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models

Ryokan Ri, Yoshimasa Tsuruoka

专题命中预训练与数据：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL

Comments ACL 2022

2109.11129 2021-09-24 cs.CL 88%

Cross-Lingual Language Model Meta-Pretraining

Zewen Chi, Heyan Huang, Luyang Liu, Yu Bai, Xian-Ling Mao

专题命中预训练与数据：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL

2109.03160 2021-09-10 cs.CL 88%

How much pretraining data do language models need to learn syntax?

Laura Pérez-Mayos, Miguel Ballesteros, Leo Wanner

专题命中预训练与数据：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL

Comments To be published in proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021)

2101.09635 2021-03-23 cs.CL 88%

WangchanBERTa: Pretraining transformer-based Thai Language Models

Lalita Lowphansirikul, Charin Polpanumas, Nawat Jantrakulchai, Sarana Nutanong

专题命中预训练与数据：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL

Comments 24 pages, edited the citation of the syllable-level tokenizer from [Chormai et al., 2020] to [Phatthiyaphaibun et al., 2020] as the authors used the syllable-level tokenizer from PyThaiNLP [Phatthiyaphaibun et al., 2020] in the experiments

2010.13826 2020-10-28 cs.CL 88%

Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining

Cheng-I Lai, Yung-Sung Chuang, Hung-Yi Lee, Shang-Wen Li, James Glass

专题命中预训练与数据：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL

2004.03720 2020-10-06 cs.CL 88%

Byte Pair Encoding is Suboptimal for Language Model Pretraining

Kaj Bostrom, Greg Durrett

专题命中预训练与数据：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL

Comments 5 pages, 3 figures. To be published in Findings of EMNLP 2020

2005.10389 2020-05-22 cs.CL 88%

Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models

Dan Iter, Kelvin Guu, Larry Lansing, Dan Jurafsky

专题命中预训练与数据：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL

Comments AC2020

1908.08788 2019-11-22 cs.IR cs.CL 88%

When Low Resource NLP Meets Unsupervised Language Model: Meta-pretraining Then Meta-learning for Few-shot Text Classification

Shumin Deng, Ningyu Zhang, Zhanlin Sun, Jiaoyan Chen, Huajun Chen

专题命中预训练与数据：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL

Comments AAAI student abstract

1901.07291 2019-01-23 cs.CL 88%

Cross-lingual Language Model Pretraining

Guillaume Lample, Alexis Conneau

专题命中预训练与数据：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL

2608.15195 2026-08-18 cs.CV 新提交 88%

Beyond Natural-Image Foundation Models: Benchmarking Satellite Pretraining for Ophthalmic Image Analysis

超越自然图像基础模型：针对眼科图像分析的卫星图像预训练基准测试

Lovre Antonio Budimir, Mingya Alexa Gong, Alyssa Foong Quinney, Ivana Matovinović, Yukun Zhou, Pearse A. Keane, Sven Lončarić, Marinko V. Šarunić

机构 * Faculty of Electrical Engineering and Computing, University of Zagreb（萨格勒布大学电气工程与计算机学院）； University College London（伦敦大学学院）； Institute of Ophthalmology, University College London（伦敦大学学院眼科研究所）； Department of Computer Science, University College London（伦敦大学学院计算机科学系）； NIHR Moorfields Biomedical Research Centre（NIHR穆尔菲尔德生物医学研究中心）； Hawkes Institute, University College London（伦敦大学学院霍克斯研究所）； Moorfields Eye Hospital NHS Foundation Trust（穆尔菲尔德眼科医院NHS基金会信托）

专题命中预训练与数据：foundation model(title,abstract)；pretraining(title,abstract)

AI总结本文针对眼科图像分析，对比卫星图像与自然图像预训练的视觉基础模型，发现卫星图像预训练在眼科任务上表现优于自然图像，部分任务可媲美医学专家模型。

Comments Accepted at the ECCV 2026 Workshop on Medical Foundation Models and Benchmarks (MEDFMB)

2409.16117 2024-09-26 eess.AS cs.SD 88%

Generative Speech Foundation Model Pretraining for High-Quality Speech Extraction and Restoration

Pin-Jui Ku, Alexander H. Liu, Roman Korostik, Sung-Feng Huang, Szu-Wei Fu, Ante Jukić

专题命中预训练与数据：pretraining(title,abstract)；foundation model(title,abstract)

Comments 5 pages, Submitted to ICASSP 2025. The implementation and configuration could be found in https://github.com/NVIDIA/NeMo/blob/main/examples/audio/conf/flow_matching_generative_ssl_pretraining.yaml The audio demo page could be found in https://kuray107.github.io/ssl_gen25-examples/index.html

2403.13430 2024-05-31 cs.CV 88%

MTP: Advancing Remote Sensing Foundation Model via Multi-Task Pretraining

Di Wang, Jing Zhang, Minqiang Xu, Lin Liu, Dongsheng Wang, Erzhong Gao, Chengxi Han, Haonan Guo, Bo Du, Dacheng Tao, Liangpei Zhang

专题命中预训练与数据：pretraining(title,abstract)；foundation model(title,abstract)

Comments Accepted by IEEE JSTARS Special issue on "Large-Scale Pretraining for Interpretation Promotion in Remote Sensing Domain". The codes and pretrained models are available at https://github.com/ViTAE-Transformer/MTP

1. 预训练与数据 12429 篇

Lingua Manga: A Generic Large Language Model Centric System for Data Curation

MISGENDERED: Limits of Large Language Models in Understanding Pronouns

From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models

Exploring New Frontiers in Agricultural NLP: Investigating the Potential of Large Language Models for Food Applications

Arithmetic-Based Pretraining -- Improving Numeracy of Pretrained Language Models

Are You Copying My Model? Protecting the Copyright of Large Language Models for EaaS via Backdoor Watermark

Emergent and Predictable Memorization in Large Language Models

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

Self-Evolution Learning for Discriminative Language Model Pretraining

Generating Faithful Synthetic Data with Large Language Models: A Case Study in Computational Social Science

Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model

Zero-Shot Listwise Document Reranking with a Large Language Model

Domain-specific Continued Pretraining of Language Models for Capturing Long Context in Mental Health

Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models

Examining Zero-Shot Vulnerability Repair with Large Language Models

Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora

On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model

DOCmT5: Document-Level Pretraining of Multilingual Language Models

Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models

Cross-Lingual Language Model Meta-Pretraining

How much pretraining data do language models need to learn syntax?

WangchanBERTa: Pretraining transformer-based Thai Language Models

Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining

Byte Pair Encoding is Suboptimal for Language Model Pretraining

Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models

When Low Resource NLP Meets Unsupervised Language Model: Meta-pretraining Then Meta-learning for Few-shot Text Classification

Cross-lingual Language Model Pretraining

Beyond Natural-Image Foundation Models: Benchmarking Satellite Pretraining for Ophthalmic Image Analysis

Generative Speech Foundation Model Pretraining for High-Quality Speech Extraction and Restoration

MTP: Advancing Remote Sensing Foundation Model via Multi-Task Pretraining