摆脱普罗克汝斯忒斯之床:用于音频语言模型的分组正交连接器
Escaping the Procrustean Bed: Groupwise Orthogonal Connectors for Audio-Language Models
AI总结:
研究音频语言模型压缩中存在的问题,提出ORCA方法,通过分组使查询输出指向不同方向,在SAKURA多跳推理中表现出色,提升分数,减少查询冗余并提高跨说话者方差。
摘要:
音频语言模型在将语音编码器的输出馈送到大型语言模型之前,通过查询变换器(Q-Former)连接器对其进行压缩。我们识别出这种压缩中的两个失败之处。连接器的输出向量坍缩到单个方向,不同说话者产生几乎无法区分的输出,诸如说话者身份、性别和韵律等副语言线索在此过程中丢失。我们的方法ORCA通过将查询分成组来扭转这种坍缩,其输出被约束指向不同方向。在SAKURA多跳推理中,与相同训练的4B基线相比,ORCA提高了26.4分,达到75.2%(8B Audio Flamingo-3为49.0%)。在连接器层面,相同变化将查询冗余减少了12倍,并将跨说话者方差提高了75倍。
英文摘要:
Audio-language models compress a speech encoder's output through a Querying Transformer (Q-Former) connector before feeding it to a large language model. We identify two failures in this compression. The connector's output vectors collapse to a single direction, and different speakers produce nearly indistinguishable outputs, with paralinguistic cues such as speaker identity, gender, and prosody lost along the way. Our method, ORCA, reverses this collapse by splitting the queries into groups whose outputs are constrained to point in different directions. On SAKURA multi-hop reasoning, ORCA gains 26.4 points over an identically trained 4B baseline, reaching 75.2% (vs. 49.0% for the 8B Audio Flamingo-3). At the connector level, the same change cuts query redundancy by 12x and raises cross-speaker variance by 75x.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。