arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2309.03340cs.CLcs.MMcs.SD
收藏

使用音频-文本共享潜在表示的参数高效且具有忠实引导的音频字幕生成

Parameter Efficient Audio Captioning With Faithful Guidance Using Audio-text Shared Latent Representation

Qualcomm Technologies(高通科技)

Arvind Krishna Sridhar, Yinyi Guo, Erik Visser, Rehana Mahfuz

AI总结:

针对自动音频字幕生成中模型过度参数化导致的幻觉和内存问题,提出数据增强检测幻觉及参数高效的推理时忠实解码算法,利用音频-文本共享潜在表示实现语义对齐,使小模型性能媲美大模型。

摘要:

在开发用于多模态到文本生成任务的预训练transformer架构方面已有大量研究。尽管性能有所提升,此类模型通常过度参数化,因此会出现幻觉和内存占用大的问题,使其难以部署在边缘设备上。本文针对自动音频字幕生成应用解决了这两个问题。首先,我们提出一种数据增强技术用于生成幻觉音频字幕,并表明基于音频-文本共享潜在空间的相似性适合检测幻觉。然后,我们提出一种参数高效的推理时忠实解码算法,使较小的音频字幕模型能达到与用更多数据训练的较大模型相当的性能。在beam decoding步骤中,较小模型利用音频-文本共享潜在表示,将生成的文本与对应输入音频进行语义对齐。通过将greedy rolled out中间beam的潜在表示投影与音频片段之间的余弦相似度纳入beam概率,引入忠实引导。我们在基准数据集上展示算法有效性,并使用常规音频字幕和语义相似性指标评估所提方案与基线的对比,同时说明性能与复杂度之间的权衡。

英文摘要:

There has been significant research on developing pretrained transformer architectures for multimodal-to-text generation tasks. Albeit performance improvements, such models are frequently overparameterized, hence suffer from hallucination and large memory footprint making them challenging to deploy on edge devices. In this paper, we address both these issues for the application of automated audio captioning. First, we propose a data augmentation technique for generating hallucinated audio captions and show that similarity based on an audio-text shared latent space is suitable for detecting hallucination. Then, we propose a parameter efficient inference time faithful decoding algorithm that enables smaller audio captioning models with performance equivalent to larger models trained with more data. During the beam decoding step, the smaller model utilizes an audio-text shared latent representation to semantically align the generated text with corresponding input audio. Faithful guidance is introduced into the beam probability by incorporating the cosine similarity between latent representation projections of greedy rolled out intermediate beams and audio clip. We show the efficacy of our algorithm on benchmark datasets and evaluate the proposed scheme against baselines using conventional audio captioning and semantic similarity metrics while illustrating tradeoffs between performance and complexity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑