Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Machine Learning Platform Department, Tencent（腾讯机器学习平台部）；School of Computer Science, Wuhan University（武汉大学计算机学院）
Ang Lv, Ruobing Xie, Xingwu Sun, Zhanhui Kang, Rui Yan
We examine the pre-training dynamics of language models, focusing on their ability to copy text from preceding context--a fundamental skill for various LLM applications, including in-context learning (ICL) and retrieval-augmented generation (RAG). We propose a novel perspective that Transformer-based language models develop copying abilities similarly to grokking, which refers to sudden generalization on test set long after the model fit to the training set. Our experiments yield three arguments: (1) The pre-training loss decreases rapidly, while the context copying ability of models initially lags and then abruptly saturates. (2) The speed of developing copying ability is independent of the number of tokens trained, similarly to how grokking speed is unaffected by dataset size as long as the data distribution is preserved. (3) Induction heads, the attention heads responsible for copying, form from shallow to deep layers during training, mirroring the development of circuits in deeper layers during grokking. We contend that the connection between grokking and context copying can provide valuable insights for more effective language model training, ultimately improving in-context performance. For example, we demonstrated that techniques that enhance grokking, such as regularization, either accelerate or enhance the development of context copying.
Copy-Paste to Mitigate Large Language Model Hallucinations
Department of Computer Science, Tianjin University of Technology（天津理工大学计算机学院）；National Institute of Health Data Science, Peking University（北京大学国家健康数据科学研究院）；Tencent Jarvis Lab（腾讯Jarvis实验室）
Investigating the Potential of Large Language Model-Based Router Multi-Agent Architectures for Foundation Design Automation: A Task Classification and Expert Selection Study
King Mongkut's University of Technology Thonburi（国王科技大学）
Performance of leading large language models in May 2025 in Membership of the Royal College of General Practitioners-style examination questions: a cross-sectional analysis
The KL3M Data Project: Copyright-Clean Training Resources for Large Language Models
Institute for the Advancement of Legal and Ethical AI (ALEA Institute)（法律与伦理AI推进研究所（ALEA研究所））；CodeX—The Stanford Center for Legal Informatics（CodeX——斯坦福法律信息学中心）；Illinois Tech—Chicago Kent College of Law（伊利诺伊理工大学芝加哥肯特法学院）；Center for Legal Technology & Data Science, Bucerius Law School（布塞留斯法学院法律技术与数据科学中心）
Repeat After Me: Transformers are Better than State Space Models at Copying
Harvard University（哈佛大学）；Kempner Institute for the Study of Natural and Artificial Intelligence（坎普纳自然与人工智能研究所）；Departments of Computer Science and Statistics（计算机科学与统计学系）
Digger: Detecting Copyright Content Mis-usage in Large Language Model Training
Beijing University of Posts and Telecommunications（北京邮电大学）；Nanyang Technological University（南洋理工大学）；Huazhong University of Science and Technology（华中科技大学）；The University of New South Wales（新南威尔士大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））
ISACL: Internal State Analyzer for Copyrighted Training Data Leakage
City University of Hong Kong（香港城市大学）；Microsoft（微软）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Google LLC（谷歌公司）；Arizona State University（亚利桑那州立大学）；Stevens Institute of Technology（史蒂文斯理工学院）
Are You Copying My Model? Protecting the Copyright of Large Language Models for EaaS via Backdoor Watermark
Beijing Jiaotong University（北京交通大学）；Microsoft Research Asia（微软亚洲研究院）；Sony AI（索尼人工智能）；Microsoft STC Asia（微软亚洲软件技术中心）；University of Science and Technology of China（中国科学技术大学）
Opaque Epistemic Mediation: How LLM Deployment Configurations Shape the Validation of Pseudo-Science
NOVA School of Science and Technology - Universidade NOVA de Lisboa（新里斯本大学新文理学院）；CIUHCT - Interuniversity Centre for the History of Science and Technology（校际科学技术史研究中心（CIUHCT））；MARE - Marine and Environmental Sciences Centre（海洋与环境科学中心（MARE））；ARNET - Aquatic Research Network Associate Laboratory（水生研究网络联合实验室（ARNET））
SoK: Large Language Model Copyright Auditing via Fingerprinting
The State Key Laboratory of Blockchain and Data Security（区块链与数据安全国家重点实验室）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）；Nanyang Technological University（南洋理工大学）；City University of Hong Kong（香港城市大学）；Sun Yat-Sen University（中山大学）
Leave No TRACE: Black-box Detection of Copyrighted Dataset Usage in Large Language Models via Watermarking
Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale Department of Computer Science Cranberry-Lemon University Pittsburgh, PA 15213, USA（计算机科学系，Cranberry-Lemon大学）；Ji Q. Ren & Yevgeny LeNet Department of Computational Neuroscience University of the Witwatersrand Joburg, South Africa（计算神经科学系，沃特瓦特斯兰大学）
