AVCap：利用视听特征作为文本标记进行字幕生成
AVCap: Leveraging Audio-Visual Features as Text Tokens for Captioning
AI总结：
本研究提出视听字幕生成框架AVCap，将视听特征作为文本标记，围绕编码器架构、预训练模型适配、模态融合效能三个维度设计，在所有指标上超越现有方法，兼具性能与扩展性优势。
摘要：
近年来，表示学习和语言模型的进步推动自动字幕生成（Automated Captioning，AC）达到新高度，可生成人类水平的描述。依托这些进展，我们提出AVCap——一种视听字幕生成框架，是适用于视听字幕生成的简单且强大的基线方法。AVCap将视听特征用作文本标记，不仅在性能上，还在模型的可扩展性和可伸缩性方面具备诸多优势。AVCap围绕三个核心维度设计：探索最优视听编码器架构、根据生成文本的特点适配预训练模型、研究模态融合在字幕生成中的效能。我们的方法在所有指标上均优于现有视听字幕生成方法，代码可在https://github.com/JongSuk1/AVCap获取。
英文摘要：
In recent years, advancements in representation learning and language models have propelled Automated Captioning (AC) to new heights, enabling the generation of human-level descriptions. Leveraging these advancements, we propose AVCap, an Audio-Visual Captioning framework, a simple yet powerful baseline approach applicable to audio-visual captioning. AVCap utilizes audio-visual features as text tokens, which has many advantages not only in performance but also in the extensibility and scalability of the model. AVCap is designed around three pivotal dimensions: the exploration of optimal audio-visual encoder architectures, the adaptation of pre-trained models according to the characteristics of generated text, and the investigation into the efficacy of modality fusion in captioning. Our method outperforms existing audio-visual captioning methods across all metrics and the code is available on https://github.com/JongSuk1/AVCap
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
