The data used to pretrain large language models has a decisive impact on a model's downstream performance, which has led to a large body of work on data selection methods that aim to automatically determine the most suitable data to use for pretraining. Existing data selection methods suffer from slow and computationally expensive processes, a problem amplified by the increasing size of models and of pretraining datasets. Data mixing, on the other hand, reduces the complexity of data selection by grouping data points together and determining sampling probabilities across entire groups. However, data mixing proportions are typically fixed before training and therefore cannot adapt to changing training dynamics. To address these limitations, we develop an efficient algorithm for Online Data Mixing (ODM) that combines elements from both data selection and data mixing. Based on multi-armed bandit algorithms, our online approach optimizes the data mixing proportions during training. Remarkably, our method trains a model that reaches the final perplexity of the next best method with 19\% fewer training iterations, and improves performance on the 5-shot MMLU benchmark by 1.9% relative accuracy, while adding negligible wall-clock time during pretraining.
University of Twente（特文特大学）；University of Oxford（牛津大学）；Bocconi University（博科尼大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；IN2P3 Computing Center / CNRS（法国国家核物理与粒子物理研究所计算中心/法国国家科学研究中心）
We study online reinforcement learning in average-reward stochastic games (SGs). An SG models a two-player zero-sum game in a Markov environment, where state transitions and one-step payoffs are determined simultaneously by a learner and an adversary. We propose the UCSG algorithm that achieves a sublinear regret compared to the game value when competing with an arbitrary opponent. This result improves previous ones under the same setting. The regret bound has a dependency on the diameter, which is an intrinsic value related to the mixing property of SGs. If we let the opponent play an optimis…
QUT（昆士兰科技大学）；UC Berkeley（加州大学伯克利分校）；University of Alberta（阿尔伯塔大学）
We study the problem of learning Markov decision processes with finite state and action spaces when the transition probability distributions and loss functions are chosen adversarially and are allowed to change with time. We introduce an algorithm whose regret with respect to any policy in a comparison class grows as the square root of the number of rounds of the game, provided the transition probabilities satisfy a uniform mixing condition. Our approach is efficient as long as the comparison class is polynomial and we can compute expectations over sample paths for each policy. Designing an ef…
Self-tuning algorithms that adapt the learning process online encourage more effective and robust learning. Among all the methods available, meta-gradients have emerged as a promising approach. They leverage the differentiability of the learning rule with respect to some hyper-parameters to adapt them in an online fashion. Although meta-gradients can be accumulated over multiple learning steps to avoid myopic updates, this is rarely used in practice. In this work, we demonstrate that whilst multi-step meta-gradients do provide a better learning signal in expectation, this comes at the cost of …
We propose the Gaussian Gated Linear Network (G-GLN), an extension to the recently proposed GLN family of deep neural networks. Instead of using backpropagation to learn features, GLNs have a distributed and local credit assignment mechanism based on optimizing a convex objective. This gives rise to many desirable properties including universality, data-efficient online learning, trivial interpretability and robustness to catastrophic forgetting. We extend the GLN framework from classification to multiple regression and density modelling by generalizing geometric mixing to a product of Gaussia…
Google Research（谷歌研究院）；Univ. Lille（里尔大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；Centrale Lille（里尔中央理工学院）；DeepMind（深度思维）；UC Berkeley（加州大学伯克利分校）；Singapore University of Technology and Design（新加坡科技设计大学）
Mean Field Games (MFGs) have been introduced to efficiently approximate games with very large populations of strategic agents. Recently, the question of learning equilibria in MFGs has gained momentum, particularly using model-free reinforcement learning (RL) methods. One limiting factor to further scale up using RL is that existing algorithms to solve MFGs require the mixing of approximated quantities such as strategies or $q$-values. This is far from being trivial in the case of non-linear function approximation that enjoy good generalization properties, e.g. neural networks. We propose two …
The multi-armed restless bandit problem is studied in the case where the pay-off distributions are stationary $φ$-mixing. This version of the problem provides a more realistic model for most real-world applications, but cannot be optimally solved in practice, since it is known to be PSPACE-hard. The objective of this paper is to characterize a sub-class of the problem where {\em good} approximate solutions can be found using tractable approaches. Specifically, it is shown that under some conditions on the $φ$-mixing coefficients, a modified version of UCB can prove effective. The main challeng…
Social networks tend to disproportionally favor connections between individuals with either similar or dissimilar characteristics. This propensity, referred to as assortative mixing or homophily, is expressed as the correlation between attribute values of nearest neighbour vertices in a graph. Recent results indicate that beyond demographic features such as age, sex and race, even psychological states such as "loneliness" can be assortative in a social network. In spite of the increasing societal importance of online social networks it is unknown whether assortative mixing of psychological sta…
Over the past decade, we have seen exponential growth in online content fueled by social media platforms. Data generation of this scale comes with the caveat of insurmountable offensive content in it. The complexity of identifying offensive content is exacerbated by the usage of multiple modalities (image, language, etc.), code-mixed language and more. Moreover, even after careful sampling and annotation of offensive content, there will always exist a significant class imbalance between offensive and non-offensive content. In this paper, we introduce a novel Code-Mixing Index (CMI) based focal…
Indian Institute of Technology Gandhinagar（印度理工学院甘地讷格尔分校）
Code-mixing is the phenomenon of using multiple languages in the same utterance of a text or speech. It is a frequently used pattern of communication on various platforms such as social media sites, online gaming, product reviews, etc. Sentiment analysis of the monolingual text is a well-studied task. Code-mixing adds to the challenge of analyzing the sentiment of the text due to the non-standard writing style. We present a candidate sentence generation and selection based approach on top of the Bi-LSTM based neural classifier to classify the Hinglish code-mixed text into one of the three sent…
Semantic homophily in online communication: evidence from Twitter
Aalto University（阿尔托大学）；University Ss. Cyril and Methodius（圣西里尔与美多德大学）；University of Tampere（坦佩雷大学）；Hong Kong University of Science and Technology（香港科技大学）；New York University（纽约大学）
Aioli: A Unified Optimization Framework for Language Model Data Mixing
Department of Computer Science, Stanford University（斯坦福大学计算机科学系）；Center for Data Science, New York University（纽约大学数据科学中心）；Computer Science Department, New York University（纽约大学计算机科学系）；Prescient Design, Genentech（基因泰克预见设计）
Persona-aware Generative Model for Code-mixed Language
Indian Institute of Technology Delhi（印度德里理工学院）；Indraprastha Institute of Information Technology Delhi（因德拉普拉斯信息技术学院德里分校）；Yardi School of Artificial Intelligence（亚迪人工智能学院）
