语言大模型 / LLM

FootGPT : A Large Language Model Development Experiment on a Minimal Setting

Eren Unlu

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Comments 10 pages, 3 figures

Large Language Models Struggle to Learn Long-Tail Knowledge

Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, Colin Raffel

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.LG

Comments ICML 2023 Camera Ready Version

Exploring the Robustness of Large Language Models for Solving Programming Problems

Atsushi Shirafuji, Yutaka Watanobe, Takumi Ito, Makoto Morishita, Yuki Nakamura, Yusuke Oda, Jun Suzuki

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Large Language Models Sometimes Generate Purely Negatively-Reinforced Text

Fabien Roger

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.LG

Comments 6 pages, 5 figures, LaTeX; added a related work section

Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuning

Mustafa Safa Ozdayi, Charith Peris, Jack FitzGerald, Christophe Dupuy, Jimit Majmudar, Haidar Khan, Rahil Parikh, Rahul Gupta

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Comments 5 pages, 3 Figures, ACL 2023

Creating a Large Language Model of a Philosopher

Eric Schwitzgebel, David Schwitzgebel, Anna Strasser

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Comments Earlier version (Nov 2, 2022) available at http://www.faculty.ucr.edu/~eschwitz/SchwitzPapers/GPT-3-Dennett-221102.pdf; stable URL for updated versions http://faculty.ucr.edu/~eschwitz/SchwitzAbs/GPT3Dennett.htm

Catch Me If You Can: Identifying Fraudulent Physician Reviews with Large Language Models Using Generative Pre-Trained Transformers

Aishwarya Deep Shukla, Laksh Agarwal, Jie Mein, Goh, Guodong, Gao, Ritu Agarwal

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Sem4SAP: Synonymous Expression Mining From Open Knowledge Graph For Language Model Synonym-Aware Pretraining

Zhouhong Gu, Sihang Jiang, Wenhao Huang, Jiaqing Liang, Hongwei Feng, Yanghua Xiao

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL、cs.AI

Large Language Models Can Be Strong Differentially Private Learners

Xuechen Li, Florian Tramèr, Percy Liang, Tatsunori Hashimoto

专题命中 预训练与数据 ：language model(title,abstract)；large language model(title)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments 31 pages; update ethics statement to clarify benefits and potential long-term harms

A Compact Pretraining Approach for Neural Language Models

Shahriar Golchin, Mihai Surdeanu, Nazgol Tavabi, Ata Kiapour

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL、cs.LG

Comments First Version

Meaning without reference in large language models

Steven T. Piantadosi, Felix Hill

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

SimVLM: Simple Visual Language Model Pretraining with Weak Supervision

Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, Yuan Cao

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL、cs.LG

Comments Published at ICLR 2022

What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?

Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, Colin Raffel

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL、cs.LG

Data Augmentation for Intent Classification with Off-the-shelf Large Language Models

Gaurav Sahu, Pau Rodriguez, Issam H. Laradji, Parmida Atighehchian, David Vazquez, Dzmitry Bahdanau

专题命中 预训练与数据 ：language model(title,abstract)；large language model(title)；prompting(abstract)；分类 cs.CL、cs.AI

Comments Accepted to 4th Workshop on NLP for Conversational AI, ACL 2022

LinkBERT: Pretraining Language Models with Document Links

Michihiro Yasunaga, Jure Leskovec, Percy Liang

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL、cs.LG

Comments Published at ACL 2022. Code, data, and pretrained models are available at https://github.com/michiyasunaga/LinkBERT

COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining

Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, Xia Song

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL、cs.LG

Comments NeurIPS 2021. (Code and Models: https://github.com/microsoft/COCO-LM)

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, Hoifung Poon

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL、cs.LG

Comments ACM Transactions on Computing for Healthcare (HEALTH)

Extracting Training Data from Large Language Models

Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, Colin Raffel

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.LG

K-XLNet: A General Method for Combining Explicit Knowledge with Language Model Pretraining

Ruiqing Yan, Lanchang Sun, Fang Wang, Xiaoming Zhang

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL、cs.AI

Knowledge-Aware Language Model Pretraining

Corby Rosset, Chenyan Xiong, Minh Phan, Xia Song, Paul Bennett, Saurabh Tiwary

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL、cs.LG

Composer Style Classification of Piano Sheet Music Images Using Language Model Pretraining

TJ Tsai, Kevin Ji

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL、cs.LG

Comments 8 pages, 7 figures. Accepted paper at the International Society for Music Information Retrieval Conference (ISMIR) 2020

Don't Stop Pretraining: Adapt Language Models to Domains and Tasks

Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, Noah A. Smith

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL、cs.LG

Comments ACL 2020

Code-switched Language Models Using Dual RNNs and Same-Source Pretraining

Saurabh Garg, Tanmay Parekh, Preethi Jyothi

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(title,abstract)；分类 cs.CL、cs.LG

Comments Accepted at EMNLP 2018

Measuring Spiritual Values and Bias of Large Language Models

Songyuan Liu, Ziyang Zhang, Runze Yan, Wei Wu, Carl Yang, Jiaying Lu

机构 * Department of Computer Science（计算机科学系） ； Center for Data Science, School of Nursing（数据科学中心、护理学院） ； Department of Religion（宗教系） ； Emory University（埃默里大学） 纠错

专题命中 预训练与数据 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL；LLM(journal_ref)

Comments 9 pages including appendix; 5 figures; 5 tables

Journal ref in Proceedings of KDD 2025 SciSoc LLM Workshop

CausalMix: Data Mixture as Causal Inference for Language Model Training

CausalMix: 数据混合作为语言模型训练的因果推断

Zinan Tang, Yukun Zhang, Shaomian Zheng, Zhuoshi Pan, Qizhi Pei, Dingnan Jin, Jun Zhou, Yujun Wang, Biqing Huang

机构 * Tsinghua University（清华大学） ； Ant Group（蚂蚁集团） ； Renmin University of China（中国人民大学） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；LLM(abstract,abstract_cn)；large language model(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 提出CausalMix框架，将数据混合优化视为因果推断问题，通过条件平均处理效应估计最优混合比例，在7B模型上提升多任务性能，并具备可解释性。

Comments 22 pages, 3 figures

HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining

HumanScale: 以自我为中心的人类视频在具身预训练中可超越真实机器人数据

Juncheng Ma, Jianxin Bi, Yufan Deng, Xuanran Zhai, Kewei Zhang, Ye Huang, Bo Liang, Shukai Gong, Jiankai Tu, Xiaotian Tang, Jiaxin Li, Kaiqi Chen, Duomin Wang, Yuqi Wang, Bingyi Kang, Eric Huang, Zhiyang Dou, Zhen Dong, Enze Xie, Wojciech Matusik, Tat-Seng Chua, Daquan Zhou

机构 * PKU（北京大学） ； NUS（新加坡国立大学） ； MIT（麻省理工学院） ； UCSB（加州大学圣塔芭芭拉分校） ； NVIDIA（英伟达） 纠错

专题命中 预训练与数据 ：pretraining(title,abstract)；large language model(abstract)；language model(abstract)；foundation model(abstract)

AI总结 本文通过系统比较发现，经过精心设计的过滤和标注流程，以自我为中心的人类视频在具身基础模型预训练中不仅可行，而且性能优于遥操作真实机器人数据，验证了“预训练于人类视频+少量机器人数据适配”的可扩展范式。

Comments Github: https://github.com/DAGroup-PKU/HumanNet/

MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understanding

MotionGPT-2：用于运动生成与理解的通用运动-语言模型

Yuan Wang, Di Huang, Yaqi Zhang, Wanli Ouyang, Jile Jiao, Xuetao Feng, Dan Xu, Shixiang Tang

机构 * Tsinghua University（清华大学） ； The University of Sydney（悉尼大学） ； University of Science and Technology of China（中国科学技术大学） ； The Chinese University of Hong Kong（香港中文大学） ； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室） ； Intime Department Store（Intime百货） ； Deepeleph ； HKUST（香港科技大学） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；LLM(abstract,abstract_cn)；large language model(abstract)；pretraining(abstract)

AI总结 提出MotionGPT-2，一种统一的大规模运动-语言模型，通过预训练大语言模型支持多模态控制条件，实现运动生成、描述和补全等多种任务，并引入Part-Aware VQVAE实现细粒度身体和手部运动表示。

DOMBA: Double Model Balancing for Access-Controlled Language Models via Minimum-Bounded Aggregation

DOMBA: 通过最小有界聚合实现访问控制语言模型的双模型平衡

Tom Segal, Asaf Shabtai, Yuval Elovici

机构 * Ben-Gurion University（本·古里安大学） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；LLM(abstract,abstract_cn)；large language model(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 提出DOMBA方法，通过最小有界平均函数聚合两个不同访问级别文档训练的语言模型的概率分布，在保证安全性的同时实现高效用。

Comments Code: https://github.com/ppo1/DOMBA 11 pages, 3 figures

Journal ref Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, pp. 25101-25109, 2025

What You Approve Is What Executes: Consent Integrity for Black-Box LLM Agents

你批准即执行：黑盒LLM代理的同意完整性

Xiaoqi Weng

专题命中 预训练与数据 ：LLM(title,title_cn)；prompting(abstract)

AI总结 针对黑盒LLM代理的批准对话框存在摘要伪造漏洞，本文引入同意完整性概念，通过可信中介确保人类看到的动作与真实执行的动作一致，并分析了该机制的局限性与权衡。

Comments Preprint. IEEE conference format. Proof-of-concept; artifact at https://github.com/zjnbwxq/agentguard-ci

Which English Do LLMs Prefer? Triangulating Structural Bias Towards American English in Foundation Models

LLMs更倾向于哪种英语？通过结构偏见的角度探讨基础模型中对美式英语的偏向

Mir Tafseer Nayeem, Davood Rafiei

机构 * University of Alberta（阿尔伯塔大学） 纠错

专题命中 预训练与数据 ：foundation model(title)；LLM(abstract)；large language model(abstract)；language model(abstract)

AI总结 本文通过分析基础模型开发各阶段的数据倾向，揭示LLMs在英语方言上的结构性偏见，发现美式英语被优先采用，引发语言同质化和全球AI部署不公的担忧。

Comments Preprint