Let $p$ denote a generative language model. Let $r$ denote a reward model that returns a scalar that captures the degree at which a draw from $p$ is preferred. The goal of language model alignment is to alter $p$ to a new distribution $ϕ$ that results in a higher expected reward while keeping $ϕ$ close to $p.$ A popular alignment method is the KL-constrained reinforcement learning (RL), which chooses a distribution $ϕ_Δ$ that maximizes $E_{ϕ_Δ} r(y)$ subject to a relative entropy constraint $KL(ϕ_Δ|| p) \leq Δ.$ Another simple alignment method is best-of-$N$, where $N$ samples are drawn from $p$ and one with highest reward is selected. In this paper, we offer a closed-form characterization of the optimal KL-constrained RL solution. We demonstrate that any alignment method that achieves a comparable trade-off between KL divergence and reward must approximate the optimal KL-constrained RL solution in terms of relative entropy. To further analyze the properties of alignment methods, we introduce two simplifying assumptions: we let the language model be memoryless, and the reward model be linear. Although these assumptions may not reflect complex real-world scenarios, they enable a precise characterization of the asymptotic behavior of both the best-of-$N$ alignment, and the KL-constrained RL method, in terms of information-theoretic quantities. We prove that the reward of the optimal KL-constrained RL solution satisfies a large deviation principle, and we fully characterize its rate function. We also show that the rate of growth of the scaled cumulants of the reward is characterized by a proper Renyi cross entropy. Finally, we show that best-of-$N$ is asymptotically equivalent to KL-constrained RL solution by proving that their expected rewards are asymptotically equal, and concluding that the two distributions must be close in KL divergence.
Distributional Vision-Language Alignment by Cauchy-Schwarz Divergence
University of Amsterdam(阿姆斯特丹大学);The Netherlands Cancer Institute(荷兰癌症研究所);Vrije Universiteit Amsterdam(阿姆斯特丹自由大学);The Arctic University of Norway(挪威北极大学);Singapore Management University(新加坡管理大学)
Beijing Academy of Artificial Intelligence(北京人工智能研究院);National University of Singapore(新加坡国立大学);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院)
Safety Alignment Depth in Large Language Models: A Markov Chain Perspective
Institute of Information Science (IIS), Academia Sinica(中央研究院资讯科学研究所);National Taiwan University(国立台湾大学);National Yang Ming Chiao Tung University(国立阳明交通大学)
LVRPO: Language-Visual Alignment with GRPO for Multimodal Understanding and Generation
Department of Machine Learning, CMU, USA(卡内基梅隆大学机器学习系,美国);Department of Machine Learning, MBZUAI, UAE(穆罕默德·本·扎耶德人工智能大学机器学习系,阿联酋);Department of Artificial Intelligence, Hanyang University ERICA, South Korea(汉阳大学ERICA校区人工智能系,韩国)
LLM Safety Alignment in Low-Resource Languages: A Systematic Literature Review
African Institute for Mathematical Sciences (AIMS)(非洲数学科学研究所);Bayero University Kano(卡诺贝罗大学);Brown University(布朗大学);Centrum Wiskunde & Informatica(数学与计算机科学中心);University of Pretoria(比勒陀利亚大学);University of Hamburg(汉堡大学);Imperial College London(伦敦帝国学院)
Cross-lingual Spoken Language Understanding with Regularized Representation Alignment
Center for Artificial Intelligence Research (CAiRE)(人工智能研究中心(CAiRE));Department of Electronic and Computer Engineering(电子与计算机工程系);The Hong Kong University of Science and Technology(香港科技大学)
尽管目前的口语语言理解系统的跨语言模型的有希望的结果,他们仍然遭受不完善的跨语言表示之间的源和目标语言,这使得性能次优的对齐。为了解决这个问题,我们提出了一种正则化方法来进一步调整跨语言的单词级和单词级表示,而无需任何外部资源。首先,我们规范化的表示用户话语的基础上,其相应的标签。其次,我们正则化潜变量模型(Liu et al.,2019)通过利用对抗性训练来解开潜在变量。跨语言语言理解任务的实验表明,我们的模型优于当前国家的最先进的方法在两个Few-Shot和zero-shot的情况下,我们的模型,训练在一个Few-Shot设置只有3\%的目标语言训练数据,达到相当的性能与所有训练数据的监督训练。