Previous language model pre-training methods have uniformly applied a next-token prediction loss to all training tokens. Challenging this norm, we posit that "9l training". Our initial analysis examines token-level training dynamics of language model, revealing distinct loss patterns for different tokens. Leveraging these insights, we introduce a new language model called Rho-1. Unlike traditional LMs that learn to predict every next token in a corpus, Rho-1 employs Selective Language Modeling (SLM), which selectively trains on useful tokens that aligned with the desired distribution. This approach involves scoring pretraining tokens using a reference model, and then training the language model with a focused loss on tokens with higher scores. When continual pretraining on 15B OpenWebMath corpus, Rho-1 yields an absolute improvement in few-shot accuracy of up to 30% in 9 math tasks. After fine-tuning, Rho-1-1B and 7B achieved state-of-the-art results of 40.6% and 51.8% on MATH dataset, respectively - matching DeepSeekMath with only 3% of the pretraining tokens. Furthermore, when continual pretraining on 80B general tokens, Rho-1 achieves 6.8% average enhancement across 15 diverse tasks, increasing both efficiency and performance of the language model pre-training.
VAST, the Valence-Assessing Semantics Test, is a novel intrinsic evaluation task for contextualized word embeddings (CWEs). VAST uses valence, the association of a word with pleasantness, to measure the correspondence of word-level LM semantics with widely used human judgments, and examines the effects of contextualization, tokenization, and LM-specific geometry. Because prior research has found that CWEs from GPT-2 perform poorly on other intrinsic evaluations, we select GPT-2 as our primary subject, and include results showing that VAST is useful for 7 other LMs, and can be used in 7 languag…
Pre-trained language models (PLMs) that use subword tokenization schemes can succeed at a variety of language tasks that require character-level information, despite lacking explicit access to the character composition of tokens. Here, studying a range of models (e.g., GPT- J, BERT, RoBERTa, GloVe), we probe what word pieces encode about character-level information by training classifiers to predict the presence or absence of a particular alphabetical character in a token, based on its embedding (e.g., probing whether the model embedding for "cat" encodes that it contains the character "a"). W…
Weighting What Matters: Boosting Sample Efficiency in Medical Report Generation via Token Reweighting
TUM School of Computation, Information and Technology（慕尼黑技术大学计算、信息与技术学院）；Technical University of Munich（慕尼黑技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Department of Computing, Imperial College London（伦敦帝国学院计算机系）
Technische Universität Darmstadt（达姆施塔特工业大学）；AI Quality & Testing Hub（AI质量与测试中心）；Hessian Center for Artificial Intelligence（黑森人工智能中心）；National Research Center for Applied Cybersecurity ATHENE（国家应用网络安全研究中心ATHENE）
What Kinds of Tokens Benefit from Distant Text? An Analysis on Long Context Language Modeling
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；School of Electronics Engineering and Computer Science, Peking University（北京大学电子与计算机工程学院）
Instituto Superior Técnico, Universidade de Lisboa（里斯本大学理工学院）；Instituto de Telecomunicações（电信研究所）；TransPerfect（TransPerfect公司）；Carnegie Mellon University（卡内基梅隆大学）
Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；Eindhoven University of Technology（埃因霍温理工大学）
Argument Unit Recognition and Classification aims at identifying argument units from text and classifying them as pro or against. One of the design choices that need to be made when developing systems for this task is what the unit of classification should be: segments of tokens or full sentences. Previous research suggests that fine-tuning language models on the token-level yields more robust results for classifying sentences compared to training on sentences directly. We reproduce the study that originally made this claim and further investigate what exactly token-based systems learned bette…
UC Santa Barbara（加州大学圣巴巴拉分校）；University of Adelaide（阿德莱德大学）；Google（谷歌）；UC Santa Cruz（加州大学圣克鲁兹分校）
Vision-and-language navigation (VLN) is a multimodal task where an agent follows natural language instructions and navigates in visual environments. Multiple setups have been proposed, and researchers apply new model architectures or training techniques to boost navigation performance. However, there still exist non-negligible gaps between machines' performance and human benchmarks. Moreover, the agents' inner mechanisms for navigation decisions remain unclear. To the best of our knowledge, how the agents perceive the multimodal input is under-studied and needs investigation. In this work, we …
What Layers When: Learning to Skip Compute in LLMs with Residual Gates
Humboldt University Berlin（洪堡大学柏林）；Qualcomm-UvA Lab, University of Amsterdam（高通-乌得勒支实验室，阿姆斯特丹大学）；FunAI Lab, University of Technology Nuremberg（FunAI实验室，纽伦堡技术大学）
Selecting What Matters: Semantic Compression-Guided Selective Pooling for Long-Context Embeddings
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；National University of Singapore（新加坡国立大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）
In incremental text to speech synthesis (iTTS), the synthesizer produces an audio output before it has access to the entire input sentence. In this paper, we study the behavior of a neural sequence-to-sequence TTS system when used in an incremental mode, i.e. when generating speech output for token n, the system has access to n + k tokens from the text sequence. We first analyze the impact of this incremental policy on the evolution of the encoder representations of token n for different values of k (the lookahead parameter). The results show that, on average, tokens travel 88% of the way to t…
What's the Difference? Supporting Users in Identifying the Effects of Prompt and Model Changes Through Token Patterns
MaiNLP, Center for Information and Language Processing, LMU Munich（MaiNLP，信息与语言处理中心，慕尼黑大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Department for Computer Vision and Machine Learning, Max Planck Institute for Informatics（计算机视觉与机器学习部门，马克斯·普朗克信息研究所）
University of Waterloo（滑铁卢大学）；Peking University（北京大学）；Capital Normal University（首都师范大学）
The semantics of a text is manifested not only by what is read, but also by what is not read. In this article, we will study how the implicit "not read" information such as end-of-paragraph (\eop) and end-of-sequence (\eos) affect the quality of text generation. Specifically, we find that the pre-trained language model GPT2 can generate better continuations by learning to generate the \eop in the fine-tuning stage. Experimental results on English story generation show that \eop can lead to higher BLEU score and lower \eos perplexity. We also conduct experiments on a self-collected Chinese essa…
