语言大模型 / LLM - arXivDaily 专题

2502.03009 2025-06-17 cs.LG cs.CL 86%

Scaling Laws for Upcycling Mixture-of-Experts Language Models

Seng Pei Liew, Takuya Kato, Sho Takase

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments ICML 2025. 16 figures, 8 tables. Code available at https://github.com/sbintuitions/sparse-upcycling-scaling-laws

2506.11485 2025-06-16 cs.CL cs.AI 86%

Relational Schemata in BERT Are Inducible, Not Emergent: A Study of Performance vs. Competence in Language Models

Cole Gawin

机构 * Cole Gawin（独立研究者）

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments 15 pages, 4 figures, 3 tables

2505.06150 2025-06-04 cs.CL cs.AI 86%

A Scaling Law for Token Efficiency in LLM Fine-Tuning Under Fixed Compute Budgets

Ryan Lagasse, Aidan Kierans, Avijit Ghosh, Shiri Dori-Hacohen

机构 * University of Connecticut Storrs（康涅狄格大学斯托尔斯分校）

专题命中预训练与数据：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

2501.00958 2025-05-14 cs.CV cs.CL cs.LG 86%

2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining

Wenqi Zhang, Hang Zhang, Xin Li, Jiashuo Sun, Yongliang Shen, Weiming Lu, Deli Zhao, Yueting Zhuang, Lidong Bing

机构 * College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）； DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）

专题命中预训练与数据：pretraining(title,abstract)；LLM(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Under review

2505.03019 2025-05-07 cs.CL cs.AI 86%

Memorization or Interpolation ? Detecting LLM Memorization through Input Perturbation Analysis

Albérick Euraste Djiré, Abdoul Kader Kaboré, Earl T. Barr, Jacques Klein, Tegawendé F. Bissyandé

机构 * ucl.ac.uk（伦敦大学学院）

专题命中预训练与数据：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

2410.12881 2025-04-28 cs.AI cs.CL 86%

MIND: Math Informed syNthetic Dialogues for Pretraining LLMs

Syeda Nahida Akter, Shrimai Prabhumoye, John Kamalu, Sanjeev Satheesh, Eric Nyberg, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro

专题命中预训练与数据：pretraining(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments 31 pages, 5 figures, 14 tables

2401.00448 2025-04-15 cs.LG cs.CL 86%

Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws

Nikhil Sardana, Jacob Portes, Sasha Doubov, Jonathan Frankle

专题命中预训练与数据：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.LG

Comments 16 pages, 7 figures, In the 41st International Conference on Machine Learning, 2024

2402.00786 2025-04-10 cs.CL cs.LG 86%

CroissantLLM: A Truly Bilingual French-English Language Model

Manuel Faysse, Patrick Fernandes, Nuno M. Guerreiro, António Loison, Duarte M. Alves, Caio Corro, Nicolas Boizard, João Alves, Ricardo Rei, Pedro H. Martins, Antoni Bigata Casademunt, François Yvon, André F. T. Martins, Gautier Viaud, Céline Hudelot, Pierre Colombo

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

2409.05816 2025-03-11 cs.CL cs.LG stat.ML 86%

Improving Pretraining Data Using Perplexity Correlations

Tristan Thrush, Christopher Potts, Tatsunori Hashimoto

专题命中预训练与数据：pretraining(title,abstract)；LLM(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments ICLR 2025

2503.01900 2025-03-05 cs.LG cs.AI 86%

LLM-Empowered Class Imbalanced Graph Prompt Learning for Online Drug Trafficking Detection

Tianyi Ma, Yiyue Qian, Zehong Wang, Zheyuan Zhang, Chuxu Zhang, Yanfang Ye

专题命中预训练与数据：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

2403.07693 2025-03-04 cs.CL cs.AI 86%

Large, Small or Both: A Novel Data Augmentation Framework Based on Language Models for Debiasing Opinion Summarization

Yanyue Zhang, Pengfei Li, Yilong Lai, Deyu Zhou, Yulan He

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；small language model(abstract)；分类 cs.CL、cs.AI

Journal ref COLING2025

2408.04665 2025-02-26 cs.CL cs.AI 86%

LLM-based MOFs Synthesis Condition Extraction using Few-Shot Demonstrations

Lei Shi, Zhimeng Liu, Yi Yang, Weize Wu, Yuyang Zhang, Hongbo Zhang, Jing Lin, Siyu Wu, Zihan Chen, Ruiming Li, Nan Wang, Zipeng Liu, Huobin Tan, Hongyi Gao, Yue Zhang, Ge Wang

专题命中预训练与数据：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

2502.08661 2025-02-18 cs.CL cs.AI 86%

Few-shot LLM Synthetic Data with Distribution Matching

Jiyuan Ren, Zhaocheng Du, Zhihao Wen, Qinglin Jia, Sunhao Dai, Chuhan Wu, Zhenhua Dong

专题命中预训练与数据：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments 10 pages, 5 figures, accepted at www 2025

2502.08145 2025-02-13 cs.LG cs.AI cs.DC 86%

Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers

Siddharth Singh, Prajwal Singhania, Aditya Ranjan, John Kirchenbauer, Jonas Geiping, Yuxin Wen, Neel Jain, Abhimanyu Hans, Manli Shu, Aditya Tomar, Tom Goldstein, Abhinav Bhatele

专题命中预训练与数据：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

2410.10450 2025-02-11 cs.AI cs.CL 86%

KBLaM: Knowledge Base augmented Language Model

Xi Wang, Taketomo Isazawa, Liana Mikaelyan, James Hensman

专题命中预训练与数据：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.AI

2404.05083 2025-02-05 cs.CV cs.CL cs.IR cs.LG 86%

DREAM: Improving Video-Text Retrieval Through Relevance-Based Augmentation Using Large Foundation Models

Yimu Wang, Shuai Yuan, Bo Xue, Xiangru Jian, Wei Pang, Mushi Wang, Ning Yu

专题命中预训练与数据：foundation model(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments NAACL 2025

2502.00894 2025-02-04 cs.CL cs.AI 86%

MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies

Ehsaneddin Asgari, Yassine El Kheir, Mohammad Ali Sadraei Javaheri

专题命中预训练与数据：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

2501.13491 2025-01-24 cs.CL cs.AI 86%

RECALL: Library-Like Behavior In Language Models is Enhanced by Self-Referencing Causal Cycles

Munachiso Nwadike, Zangir Iklassov, Toluwani Aremu, Tatsuya Hiraoka, Velibor Bojkovic, Benjamin Heinzerling, Hilal Alqaubeh, Martin Takáč, Kentaro Inui

专题命中预训练与数据：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.AI

2409.04512 2024-12-31 cs.CL cs.LG 86%

Chain-of-Translation Prompting (CoTR): A Novel Prompting Technique for Low Resource Languages

Tejas Deshpande, Nidhi Kowtal, Raviraj Joshi

专题命中预训练与数据：prompting(title,abstract)；LLM(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Accepted at PACLIC 38 (2024)

2402.01865 2024-12-11 cs.LG cs.CL stat.ML 86%

What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinement

Xisen Jin, Xiang Ren

专题命中预训练与数据：language model(title,abstract)；instruction tuning(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments ICML 2024 (Spotlight)

2412.01505 2024-12-03 cs.CL cs.LG 86%

Scaling Law for Language Models Training Considering Batch Size

Xian Shuai, Yiding Wang, Yimeng Wu, Xin Jiang, Xiaozhe Ren

专题命中预训练与数据：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.LG

2411.17135 2024-11-27 cs.AI cs.CL 86%

LLM-Based Offline Learning for Embodied Agents via Consistency-Guided Reward Ensemble

Yujeong Lee, Sangwoo Shin, Wei-Jin Park, Honguk Woo

专题命中预训练与数据：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Findings of EMNLP-2024 Camera Ready Version

2411.16985 2024-11-27 cs.CL cs.AI 86%

Teaching Smaller Language Models To Generalise To Unseen Compositional Questions (Full Thesis)

Tim Hartill

专题命中预训练与数据：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.AI

2411.07340 2024-11-13 cs.LG cs.AI 86%

Warmstarting for Scaling Language Models

Neeratyoy Mallik, Maciej Janowski, Johannes Hog, Herilalaina Rakotoarison, Aaron Klein, Josif Grabocka, Frank Hutter

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

2309.10294 2024-10-28 cs.CL cs.AI cs.SD eess.AS 86%

Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognition

Ziyang Ma, Wen Wu, Zhisheng Zheng, Yiwei Guo, Qian Chen, Shiliang Zhang, Xie Chen

专题命中预训练与数据：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments This work has been submitted to the IEEE for possible publication

2407.09985 2024-10-25 cs.AI cs.LG 86%

A Training Data Recipe to Accelerate A* Search with Language Models

Devaansh Gupta, Boyang Li

专题命中预训练与数据：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.AI、cs.LG

Comments Accepted to Findings of EMNLP, 2024

2407.05841 2024-10-23 cs.CL cs.LG 86%

An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Models

Nandini Mundra, Aditya Nanda Kishore, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, Mitesh M. Khapra

专题命中预训练与数据：language model(title,abstract)；LLM(abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments CONLL 2024 (EMNLP 2024)

2410.08196 2024-10-11 cs.CL cs.AI cs.CV 86%

MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code

Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, Hongsheng Li

专题命中预训练与数据：pretraining(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments https://github.com/mathllm/MathCoder2

2410.05355 2024-10-10 cs.CL cs.AI 86%

Falcon Mamba: The First Competitive Attention-free 7B Language Model

Jingwei Zuo, Maksim Velikanov, Dhia Eddine Rhaiem, Ilyas Chahed, Younes Belkada, Guillaume Kunsch, Hakim Hacid

专题命中预训练与数据：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.AI

2407.17467 2024-10-08 cs.CL cs.LG 86%

CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Models

Jiawei Gu, Zacc Yang, Chuanghao Ding, Rui Zhao, Fei Tan

专题命中预训练与数据：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.LG

Comments EMNLP 2024 main conference

1. 预训练与数据 12452 篇

Scaling Laws for Upcycling Mixture-of-Experts Language Models

Relational Schemata in BERT Are Inducible, Not Emergent: A Study of Performance vs. Competence in Language Models

A Scaling Law for Token Efficiency in LLM Fine-Tuning Under Fixed Compute Budgets

2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining

Memorization or Interpolation ? Detecting LLM Memorization through Input Perturbation Analysis

MIND: Math Informed syNthetic Dialogues for Pretraining LLMs

Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws

CroissantLLM: A Truly Bilingual French-English Language Model

Improving Pretraining Data Using Perplexity Correlations

LLM-Empowered Class Imbalanced Graph Prompt Learning for Online Drug Trafficking Detection

Large, Small or Both: A Novel Data Augmentation Framework Based on Language Models for Debiasing Opinion Summarization

LLM-based MOFs Synthesis Condition Extraction using Few-Shot Demonstrations

Few-shot LLM Synthetic Data with Distribution Matching

Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers

KBLaM: Knowledge Base augmented Language Model

DREAM: Improving Video-Text Retrieval Through Relevance-Based Augmentation Using Large Foundation Models

MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies

RECALL: Library-Like Behavior In Language Models is Enhanced by Self-Referencing Causal Cycles

Chain-of-Translation Prompting (CoTR): A Novel Prompting Technique for Low Resource Languages

What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinement

Scaling Law for Language Models Training Considering Batch Size

LLM-Based Offline Learning for Embodied Agents via Consistency-Guided Reward Ensemble

Teaching Smaller Language Models To Generalise To Unseen Compositional Questions (Full Thesis)

Warmstarting for Scaling Language Models

Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognition

A Training Data Recipe to Accelerate A* Search with Language Models

An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Models

MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code

Falcon Mamba: The First Competitive Attention-free 7B Language Model

CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Models