语言大模型 / LLM

When Data Is Scarce: Scaling Sparse Language Models with Repeated Training

当数据稀缺时：通过重复训练扩展稀疏语言模型

Boqian Wu, Qiao Xiao, Patrik Okanovic, Tomasz Sternal, Maurice van Keulen, Mykola Pechenizkiy, Elena Mocanu, Torsten Hoefler, Decebal Constantin Mocanu

机构 * Eindhoven University of Technology（埃因霍温理工大学） ； University of Luxembourg（卢森堡大学） ； University of Twente（埃因霍温大学） ； ETH Zürich（苏黎世联邦理工学院） 纠错

专题命中 预训练与数据 ：language model(title)；分类 cs.AI、cs.LG

AI总结 研究数据受限下稀疏训练的可扩展性，提出包含活跃参数、唯一标记、数据重复和稀疏度的缩放定律，发现稀疏训练可延迟数据饱和并改善资源权衡。

Comments Accepted at ICML2026

MambaNetBurst: Direct Byte-level Network Traffic Classification without Tokenization or Pretraining

MambaNetBurst：无需分词或预训练的直接字节级网络流量分类

Gayan K. Kulatilleke, Siamak Layeghy, Mahsa Baktashmotlagh, Marius Portmann

机构 * University of Queensland, Brisbane, Australia（昆士兰大学，布里斯班，澳大利亚） 纠错

专题命中 预训练与数据 ：pretraining(title)；分类 cs.AI、cs.LG

AI总结 MambaNetBurst基于Mamba-2架构，直接对原始字节进行分类，无需分词或预训练，实现了高效的网络流量分类。

Comments 16 pages, 2 figures. Pareto-optimal frontier. Transformer vs Mamba vs Mamba-2 scaling performance. Code and data available on request

Drift and selection in LLM text ecosystems

大语言模型文本生态系统中的漂移与选择

Søren Riis

机构 * Queen Mary University of London（伦敦玛丽女王大学） 纠错

专题命中 预训练与数据 ：LLM(title)；分类 cs.CL、cs.AI

AI总结 本文提出一个可解的数学框架，分析递归过程中的漂移与选择作用，揭示文本生态系统的稳定分布与选择过滤对结构的影响。

Not All Pretraining are Created Equal: Threshold Tuning and Class Weighting for Imbalanced Polarization Tasks in Low-Resource Settings

并非所有预训练都同等有效：低资源环境下不平衡极化任务中的阈值调节与类别加权

Abass Oguntade

机构 * African Institute of Mathematical Sciences（非洲数学科学研究所） 纠错

专题命中 预训练与数据 ：pretraining(title)；分类 cs.CL、cs.LG

AI总结 本文针对SemEval-2025极化任务，开发基于Transformer的系统，通过多语言和非洲语言专用模型、类别加权损失函数等方法，解决极化检测与分类问题，取得较高宏F1值。

Provable Ordering and Continuity in Vision-Language Pretraining for Generalizable Embodied Agents

可证明的视觉-语言排序与连续性在通用化具身体验代理中的预训练

Zhizhen Zhang, Lei Zhu, Zhen Fang, Zi Huang, Yadan Luo

机构 * The University of Queensland（昆士兰大学） ； Tongji University（同济大学） ； University of Technology Sydney（悉尼科技大学） 纠错

专题命中 预训练与数据 ：pretraining(title)；分类 cs.AI、cs.LG

AI总结 本文提出AcTOL方法，通过对比帧间语义差异和局部布朗桥约束，实现有序且连续的视觉-语言表示，提升具身体验代理在不同指令风格下的鲁棒性。

Comments NeurIPS 2025 Poster

ARCANE: A Multi-Agent Framework for Interpretable and Configurable Alignment

ARCANE：一种多智能体框架，用于可解释和可配置的对齐

Charlie Masters, Marta Grześkiewicz, Stefano V. Albrecht

专题命中 预训练与数据 ：large language model(abstract,comments)；language model(abstract,comments)；分类 cs.CL、cs.AI

AI总结 ARCANE是一种多智能体框架，通过动态规则生成实现可解释和可配置的对齐，适用于复杂长期任务。

Comments Accepted to the AAAI 2026 LLAMAS Workshop (Large Language Model Agents for Multi-Agent Systems)

What Drives Cross-lingual Ranking? Retrieval Approaches with Multilingual Language Models

什么推动跨语言排序？具有多语言语言模型的检索方法

Roksana Goworek, Olivia Macmillan-Scott, Eda B. Özyiğit

机构 * The Alan Turing Institute（艾伦·图灵研究所） ； Queen Mary University of London（伦敦女王学院） ； University College London（伦敦大学学院） 纠错

专题命中 预训练与数据 ：language model(title)；分类 cs.CL、cs.AI

AI总结 本文研究了跨语言检索中影响排序的因素，发现专门训练的密集检索模型和对比学习在低资源和跨脚本场景中表现更优，优于传统的翻译和单语言检索方法。

Gender Stereotypes in Professional Roles Among Saudis: An Analytical Study of AI-Generated Images Using Language Models

Khaloud S. AlKhalifah, Malak Mashaabi, Hend Al-Khalifa

专题命中 预训练与数据 ：language model(title)；分类 cs.CL、cs.AI

MAMUT: A Novel Framework for Modifying Mathematical Formulas for the Generation of Specialized Datasets for Language Model Training

Jonathan Drechsel, Anja Reusch, Steffen Herbold

机构 * Faculty of Computer Science and Mathematics, University of Passau, Germany（计算机科学与数学学院，帕绍大学） ； Taub Faculty for Computer Science, Technion - Israel Institute of Technology, Israel（计算机科学系，技术ion-以色列理工学院） 纠错

专题命中 预训练与数据 ：language model(title)；分类 cs.CL、cs.LG

Journal ref Transactions on Machine Learning Research (06/2025)

Geometric-Guided Few-Shot Dental Landmark Detection with Human-Centric Foundation Model

Anbang Wang, Marawan Elbatel, Keyuan Liu, Lizhuo Lin, Meng Lan, Yanqi Yang, Xiaomeng Li

机构 * Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China（电子与计算机工程系，香港科学与技术大学） ； Division of Paediatric Dentistry and Orthodontics, Faculty of Dentistry, The University of Hong Kong, Hong Kong SAR, China（牙科学院儿童牙科与正畸学系，香港大学） 纠错

专题命中 预训练与数据 ：foundation model(title)；分类 cs.AI、cs.LG

Comments MICCAI 2025

CLIP-RL: Surgical Scene Segmentation Using Contrastive Language-Vision Pretraining & Reinforcement Learning

Fatmaelzahraa Ali Ahmed, Muhammad Arsalan, Abdulaziz Al-Ali, Khalid Al-Jalham, Shidin Balakrishnan

机构 * Department of Surgery（外科部） ； Hamad Medical Corporation（哈马德医疗集团） ； Department of Computer Science and Engineering（计算机科学与工程系） ； Qatar University（卡塔尔大学） 纠错

专题命中 预训练与数据 ：pretraining(title)；分类 cs.AI、cs.LG

VietASR: Achieving Industry-level Vietnamese ASR with 50-hour labeled data and Large-Scale Speech Pretraining

Jianheng Zhuo, Yifan Yang, Yiwen Shao, Yong Xu, Dong Yu, Kai Yu, Xie Chen

机构 * School of Computer Science, MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University, shanghai innovation instituteChina Tencent AI LabUSA（计算机科学学院、人工智能国家重点实验室、上海交通大学、上海创新研究院中国腾讯AI实验室美国） 纠错

专题命中 预训练与数据 ：pretraining(title)；分类 cs.CL、cs.AI

Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo

Zachary Charles, Gabriel Teston, Lucio Dery, Keith Rush, Nova Fallen, Zachary Garrett, Arthur Szlam, Arthur Douillard

专题命中 预训练与数据 ：language model(title)；分类 cs.CL、cs.LG

Independence Tests for Language Models

Sally Zhu, Ahmed Ahmed, Rohith Kuditipudi, Percy Liang

专题命中 预训练与数据 ：language model(title)；分类 cs.CL、cs.LG

Reading Your Heart: Learning ECG Words and Sentences via Pre-training ECG Language Model

Jiarui Jin, Haoyu Wang, Hongyan Li, Jun Li, Jiahui Pan, Shenda Hong

专题命中 预训练与数据 ：language model(title)；分类 cs.AI、cs.LG

Comments 21 pages, 8 figures, accepted by International Conference on Learning Representations 2025

Resolving Discrepancies in Compute-Optimal Scaling of Language Models

Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, Yair Carmon

专题命中 预训练与数据 ：language model(title)；分类 cs.CL、cs.LG

Comments Spotlight at NeurIPS 2024

Mitigating Knowledge Conflicts in Language Model-Driven Question Answering

Han Cao, Zhaoyang Zhang, Xiangtian Li, Chufan Wu, Hansong Zhang, Wenqing Zhang

专题命中 预训练与数据 ：language model(title)；分类 cs.CL、cs.AI

Comments revised version, more figures

FGBERT: Function-Driven Pre-trained Gene Language Model for Metagenomics

ChenRui Duan, Zelin Zang, Yongjie Xu, Hang He, Zihan Liu, Siyuan Li, Zijia Song, Ju-Sheng Zheng, Stan Z. Li

专题命中 预训练与数据 ：language model(title)；分类 cs.AI、cs.LG

LDP: Generalizing to Multilingual Visual Information Extraction by Language Decoupled Pretraining

Huawen Shen, Gengluo Li, Jinwen Zhong, Yu Zhou

专题命中 预训练与数据 ：pretraining(title)；分类 cs.CL、cs.LG

Comments Accepted by AAAI2025

Open-Amp: Synthetic Data Framework for Audio Effect Foundation Models

Alec Wright, Alistair Carson, Lauri Juvela

专题命中 预训练与数据 ：foundation model(title)；分类 cs.AI、cs.LG

Looking Inward: Language Models Can Learn About Themselves by Introspection

Felix J Binder, James Chua, Tomek Korbak, Henry Sleight, John Hughes, Robert Long, Ethan Perez, Miles Turpin, Owain Evans

专题命中 预训练与数据 ：language model(title)；分类 cs.CL、cs.AI

Comments 15 pages, 9 figures

MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding

Qinzhuo Wu, Weikai Xu, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, Shuo Shang

专题命中 预训练与数据 ：language model(title)；分类 cs.CL、cs.AI

GFlowNet Pretraining with Inexpensive Rewards

Mohit Pandey, Gopeshh Subbaraj, Emmanuel Bengio

专题命中 预训练与数据 ：pretraining(title)；分类 cs.AI、cs.LG

Weakly Supervised Pretraining and Multi-Annotator Supervised Finetuning for Facial Wrinkle Detection

Ik Jun Moon, Junho Moon, Ikbeom Jang

专题命中 预训练与数据 ：pretraining(title)；分类 cs.AI、cs.LG

Single Character Perturbations Break LLM Alignment

Leon Lin, Hannah Brown, Kenji Kawaguchi, Michael Shieh

专题命中 预训练与数据 ：LLM(title)；分类 cs.CL、cs.LG

Comments 8 pages, 6 figures

Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining

Qi Zhang, Tianqi Du, Haotian Huang, Yifei Wang, Yisen Wang

专题命中 预训练与数据 ：pretraining(title)；分类 cs.CL、cs.LG

Towards Transparency: Exploring LLM Trainings Datasets through Visual Topic Modeling and Semantic Frame

Charles de Dampierre, Andrei Mogoutov, Nicolas Baumard

专题命中 预训练与数据 ：LLM(title)；分类 cs.CL、cs.AI

ChatLang-8: An LLM-Based Synthetic Data Generation Framework for Grammatical Error Correction

Jeiyoon Park, Chanjun Park, Heuiseok Lim

专题命中 预训练与数据 ：LLM(title)；分类 cs.CL、cs.AI

Comments preprint

Federated Generative Learning with Foundation Models

Jie Zhang, Xiaohua Qi, Bo Zhao

专题命中 预训练与数据 ：foundation model(title)；分类 cs.AI、cs.LG

DiffRoll: Diffusion-based Generative Music Transcription with Unsupervised Pretraining Capability

Kin Wai Cheuk, Ryosuke Sawata, Toshimitsu Uesaka, Naoki Murata, Naoya Takahashi, Shusuke Takahashi, Dorien Herremans, Yuki Mitsufuji

专题命中 预训练与数据 ：pretraining(title)；分类 cs.AI、cs.LG

Journal ref Proceedings of ICASSP - IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023