逐步融合：一种面向鲁棒且感知指令的ASR与OCR的大语言模型生成式融合解码算法
Let's Fuse Step by Step: A Generative Fusion Decoding Algorithm with LLMs for Robust and Instruction-Aware ASR and OCR
AI总结：
本研究提出即插即用的生成式融合解码（GFD）框架，通过字节级似然计算适配不同模型词元空间，将LLM集成到ASR与OCR系统中，在中英基准上优于级联方法，还可迁移LLM上下文学习能力，实现感知指令的自适应ASR，WER最高降17.7%。
摘要：
我们提出了“生成式融合解码”（Generative Fusion Decoding，GFD），这是一种新颖的浅层融合框架，旨在将大语言模型（LLM）集成到自动语音识别（ASR）和光学字符识别（OCR）的跨模态文本识别系统中。我们推导了必要的公式，通过在字节级别计算似然，使GFD能够在不同模型的不匹配词元空间中运行，从而实现解码过程中的无缝融合与同步推进。GFD采用即插即用设计，无需任何重新训练即可兼容各类自回归模型。通过与LLM的中间层频繁交互，GFD在通用ASR和OCR任务中表现出色，在英文和中文基准测试中优于级联方法。此外，GFD迁移了LLM的上下文学习能力，支持感知指令和长上下文场景下的自适应ASR，词错误率（WER）显著降低高达17.7%。
英文摘要：
We propose "Generative Fusion Decoding" (GFD), a novel shallow fusion framework designed to integrate large language models (LLMs) into cross-modal text recognition systems for automatic speech recognition (ASR) and optical character recognition (OCR). We derive the necessary formulations to enable GFD to operate across mismatched token spaces of different models by calculating likelihood at the byte level, thereby enabling seamless fusion and synchronous progression during the decoding process. GFD is plug-and-play by design, making it readily compatible with various auto-regressive models without the need for any re-training. GFD proves effective for general ASR and OCR tasks through intermediate and frequent interactions with LLMs, surpassing cascaded methods in English and Mandarin benchmarks. In addition, GFD transfers in-context learning abilities of LLMs and allows for adaptive ASR in instruction-aware and long-context settings, yielding significant WER reductions of up to 17.7\%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
