语言大模型 / LLM

Beyond Public Access in LLM Pre-Training Data

Sruly Rosenblat, Tim O'Reilly, Ilan Strauss

机构 * AI Disclosures Project, Code for Science and Society（AI披露项目，代码用于社会） ； O’Reilly Media（O’Reilly媒体） ； Institute for Innovation and Public Purpose, University College London（创新与公共目的研究所，伦敦大学学院） 纠错

专题命中 预训练与数据 ：LLM(title)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Supervised Pretraining for Material Property Prediction

Chowdhury Mohammad Abid Rahman, Aldo H. Romero, Prashnna K. Gyawali

机构 * Lane Department of Computer Science and Electrical Engineering, West Virginia University（兰恩计算机科学与电气工程系，西弗吉尼亚大学） ； Department of Physics and Astronomy, West Virginia University（物理与天文学系，西弗吉尼亚大学） 纠错

专题命中 预训练与数据 ：pretraining(title,abstract)；foundation model(abstract)；分类 cs.AI、cs.LG

Comments 21 pages, 7 figures, 2 algorithms, 6 tables

DP2FL: Dual Prompt Personalized Federated Learning in Foundation Models

Ying Chang, Xiaohu Shi, Xiaohui Zhao, Zhaohuang Chen, Deyin Ma

机构 * College of Software, Jilin University（吉林大学软件学院） ； College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院） ； College of Computer Science and Engineering, Changchun University of Technology（长春理工大学计算机科学与工程学院） 纠错

专题命中 预训练与数据 ：foundation model(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

DataComp-LM: In search of the next generation of training sets for language models

Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, Vaishaal Shankar

机构 * University of Washington（华盛顿大学） ； Apple（苹果公司） ； Toyota Research Institute（丰田研究院） ； UT Austin（得克萨斯大学） ； Tel Aviv University（特拉维夫大学） ； Columbia University（哥伦比亚大学） ； Stanford（斯坦福） ； UCLA（加州大学洛杉矶分校） ； JSC ； LAION ； AI2 ； TUM（慕尼黑技术大学） ； CMU（卡内基梅隆大学） ； Hebrew University（希伯来大学） ； SambaNova ； Cornell（康奈尔大学） ； USC（南加州大学） ； Harvard（哈佛大学） ； UCSB（加州大学圣塔芭芭拉分校） ； SynthLabs ； Contextual AI ； DatologyAI 纠错

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Project page: https://www.datacomp.ai/dclm/

No Need to Talk: Asynchronous Mixture of Language Models

Anastasiia Filippova, Angelos Katharopoulos, David Grangier, Ronan Collobert

专题命中 预训练与数据 ：language model(title,abstract)；LLM(abstract)；分类 cs.CL、cs.LG

Comments 23 pages

From Tokens to Lattices: Emergent Lattice Structures in Language Models

Bo Xiong, Steffen Staab

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments ICLR 2025

Fast Adaptation with Behavioral Foundation Models

Harshit Sikchi, Andrea Tirinzoni, Ahmed Touati, Yingchen Xu, Anssi Kanervisto, Scott Niekum, Amy Zhang, Alessandro Lazaric, Matteo Pirotta

专题命中 预训练与数据 ：foundation model(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments 25 pages

Hide and Seek in Noise Labels: Noise-Robust Collaborative Active Learning with LLM-Powered Assistance

Bo Yuan, Yulin Chen, Yin Zhang, Wei Jiang

专题命中 预训练与数据 ：LLM(title)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models

Muchao Ye, Weiyang Liu, Pan He

专题命中 预训练与数据 ：language model(title,abstract)；instruction tuning(abstract)；分类 cs.AI、cs.LG

Comments Accepted in CVPR 2025

Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling

Haebin Shin, Lei Ji, Xiao Liu, Yeyun Gong

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Not All LLM-Generated Data Are Equal: Rethinking Data Weighting in Text Classification

Hsun-Yu Kuo, Yin-Hsiang Liao, Yu-Chieh Chao, Wei-Yun Ma, Pu-Jen Cheng

专题命中 预训练与数据 ：LLM(title)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments ICLR 2025 camera ready

Specialized Foundation Models Struggle to Beat Supervised Baselines

Zongzhe Xu, Ritvik Gupta, Wenduo Cheng, Alexander Shen, Junhong Shen, Ameet Talwalkar, Mikhail Khodak

专题命中 预训练与数据 ：foundation model(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments The first two authors contributed equally. The order was determined by coin flip

Command R7B Arabic: A Small, Enterprise Focused, Multilingual, and Culturally Aware Arabic LLM

Yazeed Alnumay, Alexandre Barbet, Anna Bialas, William Darling, Shaan Desai, Joan Devassy, Kyle Duffy, Stephanie Howe, Olivia Lasche, Justin Lee, Anirudh Shrinivason, Jennifer Tracey

专题命中 预训练与数据 ：LLM(title)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

The Power of LLM-Generated Synthetic Data for Stance Detection in Online Political Discussions

Stefan Sylvius Wagner, Maike Behrendt, Marc Ziegele, Stefan Harmeling

专题命中 预训练与数据 ：LLM(title,abstract)；prompting(abstract)；分类 cs.CL、cs.LG

Comments ICLR 2025 Spotlight

Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling

David Grangier, Simin Fan, Skyler Seto, Pierre Ablin

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments 23 pages, presented at the International Conference on Learning Representation (ICLR), 2025

Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation

Nachiket Kotalwar, Alkis Gotovos, Adish Singla

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.AI、cs.LG

Scaling LLM Pre-training with Vocabulary Curriculum

Fangyuan Yu

专题命中 预训练与数据 ：LLM(title)；language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments under review

Krutrim LLM: Multilingual Foundational Model for over a Billion People

Aditya Kallappa, Palash Kamble, Abhinav Ravi, Akshat Patidar, Vinayak Dhruv, Deepak Kumar, Raghav Awasthi, Arveti Manjunath, Himanshu Gupta, Shubham Agarwal, Kumar Ashish, Gautam Bhargava, Chandra Khatri

专题命中 预训练与数据 ：LLM(title,abstract)；foundation model(abstract)；分类 cs.CL、cs.AI

Mitigating Data Scarcity in Time Series Analysis: A Foundation Model with Series-Symbol Data Generation

Wenxuan Wang, Kai Wu, Yujian Betterest Li, Dan Wang, Xiaoyu Zhang, Jing Liu

专题命中 预训练与数据 ：foundation model(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Transformers and Their Roles as Time Series Foundation Models

Dennis Wu, Yihan He, Yuan Cao, Jianqing Fan, Han Liu

专题命中 预训练与数据 ：foundation model(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments 34 Pages, 2 Figures

COS(M+O)S: Curiosity and RL-Enhanced MCTS for Exploring Story Space via Language Models

Tobias Materzok

专题命中 预训练与数据 ：language model(title,abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI

RegMix: Data Mixture as Regression for Language Model Pre-training

Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, Min Lin

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments ICLR 2025

Self-supervised video pretraining yields robust and more human-aligned visual representations

Nikhil Parthasarathy, S. M. Ali Eslami, João Carreira, Olivier J. Hénaff

专题命中 预训练与数据 ：pretraining(title,abstract)；foundation model(abstract)；分类 cs.AI、cs.LG

Comments Accepted to 37th Conference on Neural Information Processing Systems (NeurIPS 2023)

Whose Morality Do They Speak? Unraveling Cultural Bias in Multilingual Language Models

Meltem Aksoy

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Applying LLM and Topic Modelling in Psychotherapeutic Contexts

Alexander Vanin, Vadim Bolshev, Anastasia Panfilova

专题命中 预训练与数据 ：LLM(title)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments 18 pages, 4 figures

LLMCL-GEC: Advancing Grammatical Error Correction with LLM-Driven Curriculum Learning

Tao Fang, Derek F. Wong, Lusheng Zhang, Keyan Jin, Qiang Zhang, Tianjiao Li, Jinlong Hou, Lidia S. Chao

专题命中 预训练与数据 ：LLM(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Derek F. Wong is the corresponding author. The preprint version consists of 15 Pages, 5 Figures, 5 Tables, and 3 Appendices

NoteContrast: Contrastive Language-Diagnostic Pretraining for Medical Text

Prajwal Kailas, Max Homilius, Rahul C. Deo, Calum A. MacRae

专题命中 预训练与数据 ：pretraining(title)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Journal ref Proceedings of the 3rd Machine Learning for Health Symposium, PMLR 225:201-216, 2023

HITgram: A Platform for Experimenting with n-gram Language Models

Shibaranjani Dasgupta, Chandan Maity, Somdip Mukherjee, Rohan Singh, Diptendu Dutta, Debasish Jana

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

SOLO: A Single Transformer for Scalable Vision-Language Modeling

Yangyi Chen, Xingyao Wang, Hao Peng, Heng Ji

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

Comments Accepted to TMLR

Seeing Syntax: Uncovering Syntactic Learning Limitations in Vision-Language Models

Sri Harsha Dumpala, David Arps, Sageev Oore, Laura Kallmeyer, Hassan Sajjad

专题命中 预训练与数据 ：language model(title,abstract)；foundation model(abstract)；分类 cs.CL、cs.LG