Co-evolutionary self-play, where one language model generates problems and another solves them, promises curriculum learning without human supervision. The promise breaks down early in practice. The proposer converges to a narrow distribution of problems that satisfy the reward function, and the collapsed curriculum teaches the solver little, stalling the loop. We introduce vocabulary dropout, a lightweight intervention that randomly masks the proposer's output logits during both policy training and curriculum generation. The mask is hard and non-stationary, so the proposer cannot lock into fixed token sequences. Training Qwen3-4B and Qwen3-8B on mathematical reasoning via R-Zero, vocabulary dropout sustains proposer diversity throughout training across lexical, semantic, and functional measures, and improves the solver by an average of +4.4 points at 8B with the largest gains on competition-level benchmarks. Explicit action-space constraints, filling the structural role that game rules fill in classical self-play, can keep co-evolution in language productive. Vocabulary dropout is one simple way to impose them.
Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Data
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）；Tsinghua University（清华大学）；WeChat, Tencent（微信、腾讯）；Department of Data Science, City University of Hong Kong（香港城市大学数据科学系）
This paper carries out an empirical analysis of various dropout techniques for language modelling, such as Bernoulli dropout, Gaussian dropout, Curriculum Dropout, Variational Dropout and Concrete Dropout. Moreover, we propose an extension of variational dropout to concrete dropout and curriculum dropout with varying schedules. We find these extensions to perform well when compared to standard dropout approaches, particularly variational curriculum dropout with a linear schedule. Largest performance increases are made when applying dropout on the decoder layer. Lastly, we analyze where most of…
Computer Laboratory, University of Cambridge（剑桥大学计算机实验室）；Catalyst AI Ltd.（Catalyst AI 有限公司）
The structure of curriculum plays a vital role in our learning process, both as children and adults. Presenting material in ascending order of difficulty that also exploits prior knowledge can have a significant impact on the rate of learning. However, the notion of difficulty and prior knowledge differs from person to person. Motivated by the need for a personalised curriculum, we present a novel method of curriculum learning for vocabulary words in the form of visual prompts. We employ a reinforcement learning model grounded in pedagogical theories that emulates the actions of a tutor. We si…
An NLP-Driven Framework for Curriculum-Labor Market Alignment: Schema-Constrained LLM Extraction, ESCO-Anchored Semantic Matching, and Multi-Dimensional Gap Quantification
United Arab Emirates University（阿联酋大学）；Abu Dhabi Polytechnic（阿布扎比理工学院）
Moscow Institute of Physics and Technology（莫斯科物理技术学院）；National Research University Higher School of Economics（国家研究型高等经济大学）；University of Edinburgh（爱丁堡大学）；University of Amsterdam（阿姆斯特丹大学）；Yandex（Yandex公司）
Subword segmentation is widely used to address the open vocabulary problem in machine translation. The dominant approach to subword segmentation is Byte Pair Encoding (BPE), which keeps the most frequent words intact while splitting the rare ones into multiple tokens. While multiple segmentations are possible even with the same vocabulary, BPE splits words into unique sequences; this may prevent a model from better learning the compositionality of words and being robust to segmentation errors. So far, the only way to overcome this BPE imperfection, its deterministic nature, was to create anoth…
SegTME-UNI2: A Foundation Model-Based Framework for Generalisable Multiclass Cell Segmentation and LLM-Driven Tumour Microenvironment Characterisation in Histopathology
Department of Data Science and Artificial Intelligence, School of Computing and Artificial Intelligence, Faculty of Engineering and Technology, Sunway University（双威大学工程与技术学院计算与人工智能学院数据科学与人工智能系）；Faculty of Dentistry, Universiti Malaya（马来亚大学牙科学院）
