跨模态的分类器引导字幕生成
Classifier-Guided Captioning Across Modalities
AI总结：
本文提出一种推理期文本分类器引导框架，利用 GPT-4 生成数据训练分类器，使冻结字幕模型适配音频等跨模态语义，并在零样本音频字幕生成上达到 SOTA。
摘要：
当前大多数字幕生成系统使用在特定场景数据上训练的语言模型，例如通过 Amazon Mechanical Turk 获取的基于图像的字幕数据，这限制了它们泛化到其他模态分布和语境的能力。这一局限阻碍了其在音频或视频字幕生成等任务上的表现，因为这些任务需要不同的语义线索。应对这一挑战，对于构建可适用于多种真实世界场景、更具适应性和通用性的字幕生成框架至关重要。在本研究中，我们提出一种方法，使字幕生成网络能够适应其他场景的语义，例如在音频字幕生成中捕捉可听性；在该任务中，描述声音及其来源至关重要。我们的框架包含两个主要组件：(i) 一个融合语言模型（LM）的冻结字幕生成系统，以及 (ii) 一个引导该字幕生成系统的文本分类器。该分类器在由 GPT-4 自动生成的数据集上训练，并使用专门设计的定制提示词，以增强所生成字幕的关键方面。重要的是，该框架仅在推理阶段运行，无需对底层字幕生成模型进行进一步训练。我们在多种模型和模态上评估了该框架，并重点关注音频字幕生成，报告了富有前景的结果。值得注意的是，当与现有零样本音频字幕生成系统结合时，我们的框架提升了其质量，并在零样本音频字幕生成中取得了最先进性能。
英文摘要：
Most current captioning systems use language models trained on data from specific settings, such as image-based captioning via Amazon Mechanical Turk, limiting their ability to generalize to other modality distributions and contexts. This limitation hinders performance in tasks like audio or video captioning, where different semantic cues are needed. Addressing this challenge is crucial for creating more adaptable and versatile captioning frameworks applicable across diverse real-world contexts. In this work, we introduce a method to adapt captioning networks to the semantics of alternative settings, such as capturing audibility in audio captioning, where it is crucial to describe sounds and their sources. Our framework consists of two main components: (i) a frozen captioning system incorporating a language model (LM), and (ii) a text classifier that guides the captioning system. The classifier is trained on a dataset automatically generated by GPT-4, using tailored prompts specifically designed to enhance key aspects of the generated captions. Importantly, the framework operates solely during inference, eliminating the need for further training of the underlying captioning model. We evaluate the framework on various models and modalities, with a focus on audio captioning, and report promising results. Notably, when combined with an existing zero-shot audio captioning system, our framework improves its quality and sets state-of-the-art performance in zero-shot audio captioning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
