语音翻译的思维链提示
Chain-of-Thought Prompting for Speech Translation
AI总结：
提出利用ASR转录作为提示，通过类似思维链的两步过程引导语音翻译，采用LoRA适配T5，在6个AST任务上平均提升2.4 BLEU点。
摘要：
大型语言模型（LLMs）在语言理解和生成方面展现了显著的进步。基于文本LLMs的成功，最近的研究将这些模型适配为使用语音嵌入进行提示，产生了在自动语音识别（ASR）和自动语音翻译（AST）中表现出色的Speech-LLM模型。在这项工作中，我们提出了一种新颖的方法，利用ASR转录作为基于编码器-解码器文本LLM构建的Speech-LLM中AST的提示。该Speech-LLM模型由一个语音编码器和一个编码器-解码器结构的Megatron-T5组成。通过首先解码语音以生成ASR转录，随后使用这些转录以及编码后的语音进行提示，我们以类似思维链（CoT）提示的两步过程引导语音翻译。采用低秩适配（LoRA）对T5 LLM进行模型适配，并显示出优于全模型微调的性能。实验结果表明，所提出的CoT提示显著提升了AST性能，在6个En->X或X->En AST任务中，与仅使用语音提示相比，平均提高了2.4个BLEU点。此外，与一种预测ASR和AST转录串联序列的相关CoT预测方法相比，我们的方法平均高出2个BLEU点。
英文摘要：
Large language models (LLMs) have demonstrated remarkable advancements in language understanding and generation. Building on the success of text-based LLMs, recent research has adapted these models to use speech embeddings for prompting, resulting in Speech-LLM models that exhibit strong performance in automatic speech recognition (ASR) and automatic speech translation (AST). In this work, we propose a novel approach to leverage ASR transcripts as prompts for AST in a Speech-LLM built on an encoder-decoder text LLM. The Speech-LLM model consists of a speech encoder and an encoder-decoder structure Megatron-T5. By first decoding speech to generate ASR transcripts and subsequently using these transcripts along with encoded speech for prompting, we guide the speech translation in a two-step process like chain-of-thought (CoT) prompting. Low-rank adaptation (LoRA) is used for the T5 LLM for model adaptation and shows superior performance to full model fine-tuning. Experimental results show that the proposed CoT prompting significantly improves AST performance, achieving an average increase of 2.4 BLEU points across 6 En->X or X->En AST tasks compared to speech prompting alone. Additionally, compared to a related CoT prediction method that predicts a concatenated sequence of ASR and AST transcripts, our method performs better by an average of 2 BLEU points.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
