arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.06211cs.CLcs.AIeess.AS
收藏

LF²AR:考虑分层动态以改进语言模型的多模态适配

LF${}^{2}$AR: Accounting for Layerwise Dynamics to Improve Multimodal Adaptation of Language Models

Université de Toulon, Aix-Marseille Université, CNRS, LIS, France(法国图卢兹大学、马赛大学、CNRS、LIS);Department of Engineering, University of Cambridge, UK(剑桥大学工程系);Mitsubishi Electric Research Laboratories (MERL), Cambridge, MA, USA(三菱电机研究实验室(MERL));LIA, Avignon Université, France(法国阿维尼翁大学LIA);LIUM, Le Mans Université, France(法国勒芒大学LIUM);Zenidoc, Marseille, France(法国马赛Zenidoc)

Santiago Cuervo, Adel Moumen, Yanis Labrak, Sameer Khurana, Antoine Laurent, Mickael Rouvier, Phil Woodland, Ricard Marxer

AI总结:

本研究提出LF²AR架构,通过分层抽象-细化动态设计适配机制,在文本转图像、语音模态上提升语言模型性能,支持1.9倍生成加速。

摘要:

文本预训练的语言模型(LMs)编码了丰富的世界知识,但将其适配以处理和生成音频、图像等感知模态,同时有效利用该知识仍是一项挑战。感知模态比文本更细粒度、语义密度更低,因此尚不明确文本预训练期间学习的函数应如何复用。本研究通过Transformer语言模型中观察到的分层抽象-细化动态视角探讨该问题:表示首先变得更抽象和组合,随后被细化为可预测细粒度结构的表示。该视角表明,将语言模型适配到更细粒度的模态需要:(i)在输入侧分配额外的细到粗处理,在输出侧分配粗到细处理,这与后期模态融合及我们称为后期裂变的输出侧类似机制一致;(ii)允许输出预测器保留依赖输入的对高级语义结构和低级感知细节的选择性访问,这促使我们在裂变中使用注意力残差。我们将该观点实例化为LF²AR,一种结合这些机制的简单架构,并在文本转图像、语音这两种可控制与文本语义对应关系的模态上进行研究。在参数规模从1.35亿到20亿的模型上适配这些模态后,我们发现这些组件可增加特征抽象度、增强跨模态对齐、使这种对齐在更小的计算预算下即可出现、提升类文本预测结构的保留能力,并在文本转图像和语音版本的语言理解与推理基准上取得更好性能。此外,注意力残差会诱导对主干深层的稀疏、可解释使用,支持提前退出解码,实现1.9倍的生成加速。

英文摘要:

Text-pretrained language models (LMs) encode rich world knowledge, but adapting them to process and generate perceptual modalities such as audio and images while effectively leveraging that knowledge remains challenging. Perceptual modalities are finer-grained and less semantically dense than text, making it unclear how functions learned during text pretraining can be reused. We study this problem through the lens of a layerwise abstraction-refinement dynamic observed in transformer LMs: representations first become more abstract and compositional, then are refined into representations predictive of fine-grained structure. This perspective suggests that adapting an LM to finer-grained modalities requires: (i) allocating additional fine-to-coarse processing at the input and coarse-to-fine processing at the output, consistent with late modality fusion and an output-side analogue we term late fission; and (ii) allowing the output predictor to preserve input-dependent selective access to both high-level semantic structure and low-level perceptual detail, motivating our use of attention residuals in fission. We instantiate this view in LF${}^{2}$AR, a simple architecture combining these mechanisms, and study it on text-as-images and speech, two modalities for which semantic correspondence to text can be controlled. Across models ranging from 135M to 2B parameters and adapted to these modalities, we find that these components increase feature abstraction, strengthen cross-modal alignment, enable such alignment to emerge at smaller compute budgets, improve preservation of text-like predictive structure, and yield better performance on text-as-images and speech versions of language understanding and reasoning benchmarks. Additionally, attention residuals induce sparse, interpretable use of deep backbone layers, enabling early-exit decoding with a 1.9$\times$ generation speedup.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑