The composition of training data mixtures is critical for effectively training large language models (LLMs), as it directly impacts their performance on downstream tasks. Our goal is to identify an optimal data mixture to specialize an LLM for a specific task with access to only a few examples. Traditional approaches to this problem include ad-hoc reweighting methods, importance sampling, and gradient alignment techniques. This paper focuses on gradient alignment and introduces Dynamic Gradient Alignment (DGA), a scalable online gradient alignment algorithm. DGA dynamically estimates the pre-training data mixture on which the models' gradients align as well as possible with those of the model on the specific task. DGA is the first gradient alignment approach that incurs minimal overhead compared to standard pre-training and outputs a competitive model, eliminating the need for retraining the model. Experimentally, we demonstrate significant improvements over importance sampling in two key scenarios: (i) when the pre-training set is small and importance sampling overfits due to limited data; and (ii) when there is insufficient specialized data, trapping importance sampling on narrow pockets of data. Our findings underscore the effectiveness of gradient alignment methods in optimizing training data mixtures, particularly in data-constrained environments, and offer a practical solution for enhancing LLM performance on specific tasks with limited data availability.
PMoL: Parameter Efficient MoE for Preference Mixing of LLM Alignment
Harbin Institute of Technology（哈尔滨工业大学）；Beijing Normal University - Hong Kong Baptist University United International College（北京师范大学-香港浸会大学联合国际学院）；Qilu University of Technology（齐鲁工业大学）
Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization
School of Computer Science and Engineering, Southeast University, Nanjing, China（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及其交叉应用国家重点实验室）
MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment
Department of Statistics, Rutgers University, New Brunswick, United States；Department of Computer Science, Rutgers University, New Brunswick, United States；College of Management of Technology, EPFL, Switzerland；Department of Computer Science, ETH Zurich, Switzerland
Online alignment in machine translation refers to the task of aligning a target word to a source word when the target sequence has only been partially decoded. Good online alignments facilitate important applications such as lexically constrained translation where user-defined dictionaries are used to inject lexical constraints into the translation model. We propose a novel posterior alignment technique that is truly online in its execution and superior in terms of alignment error rates compared to existing methods. Our proposed inference technique jointly considers alignment and token probabi…
Recurrent neural network models with an attention mechanism have proven to be extremely effective on a wide variety of sequence-to-sequence problems. However, the fact that soft attention mechanisms perform a pass over the entire input sequence when producing each element in the output sequence precludes their use in online settings and results in a quadratic time complexity. Based on the insight that the alignment between input and output sequence elements is monotonic in many problems of interest, we propose an end-to-end differentiable method for learning monotonic alignments which, at test…
Institute for Interdisciplinary Information Sciences（交叉信息研究院）；Tsinghua University（清华大学）；Google Brain（谷歌大脑）；Open AI（OpenAI）
Sequence-to-sequence models with soft attention had significant success in machine translation, speech recognition, and question answering. Though capable and easy to use, they require that the entirety of the input sequence is available at the beginning of inference, an assumption that is not valid for instantaneous translation and speech recognition. To address this problem, we present a new method for solving sequence-to-sequence problems using hard online alignments instead of soft offline alignments. The online alignments model is able to start producing outputs without the need to first …
This article describes an efficient training method for online streaming attention-based encoder-decoder (AED) automatic speech recognition (ASR) systems. AED models have achieved competitive performance in offline scenarios by jointly optimizing all components. They have recently been extended to an online streaming framework via models such as monotonic chunkwise attention (MoChA). However, the elaborate attention calculation process is not robust for long-form speech utterances. Moreover, the sequence-level training objective and time-restricted streaming encoder cause a nonnegligible delay…
One-Step Bellman Alignment Enables Provably Efficient Transfer in Online RL
Department of Technology, Operations, & Statistics, New York University（纽约大学技术、运营与统计系）；Department of Operations Research & Financial Engineering, Princeton University（普林斯顿大学运筹学与金融工程系）；Department of Computer Science, Dartmouth College（达特茅斯学院计算机科学系）
Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）；PhiGent Robotics（鉴智机器人）
Data mixing strategies (e.g., CutMix) have shown the ability to greatly improve the performance of convolutional neural networks (CNNs). They mix two images as inputs for training and assign them with a mixed label with the same ratio. While they are shown effective for vision transformers (ViTs), we identify a token fluctuation phenomenon that has suppressed the potential of data mixing strategies. We empirically observe that the contributions of input tokens fluctuate as forward propagating, which might induce a different mixing ratio in the output tokens. The training target computed by the…
Online Adaptation via Dual-Stage Alignment and Self-Supervision for Fast-Calibration Brain-Computer Interfaces
School of Information, Shanxi University of Finance and Economics（山西财经大学信息学院）；Chinese Academy of Sciences（中国科学院）；State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation（多模态人工智能系统国家重点实验室，自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）；Joint Laboratory of Intelligence Science and Technology, Institute of Systems Engineering, Macau University of Science and Technology（澳门科技大学智能科学与技术联合实验室）
University of Twente（特文特大学）；University of Oxford（牛津大学）；Bocconi University（博科尼大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；IN2P3 Computing Center / CNRS（法国国家核物理与粒子物理研究所计算中心/法国国家科学研究中心）
STAG: Structural Test-time Alignment of Gradients for Online Adaptation
Seoul National University, Seoul, Republic of Korea；Division of Artificial Intelligence \& Software, Ewha Womans University, Seoul, Republic of Korea；Microsoft, Redmond, WA, USA；Georgia Institute of Technology, Atlanta, GA, USA
