The NTT DCASE2020 Challenge Task 6 system: Automated Audio Captioning with Keywords and Sentence Length Estimation

Improving the Performance of Automated Audio Captioning via Integrating the Acoustic and Semantic Information

Automated audio captioning (AAC) has developed rapidly in recent years, involving acoustic signal processing and natural language processing to generate human-readable sentences for audio clips. The current models are generally based on the neural encoder-decoder architecture, and their decoder mainly uses acoustic information that is extracted from the CNN-based encoder. However, they have ignored semantic information that could help the AAC model to generate meaningful descriptions. This paper proposes a novel approach for automated audio captioning based on incorporating semantic and acoust…

Audio Captioning with Composition of Acoustic and Semantic Information

Generating audio captions is a new research area that combines audio and natural language processing to create meaningful textual descriptions for audio clips. To address this problem, previous studies mostly use the encoder-decoder based models without considering semantic information. To fill this gap, we present a novel encoder-decoder architecture using bi-directional Gated Recurrent Units (BiGRU) with audio and semantic embeddings. We extract semantic embedding by obtaining subjects and verbs from the audio clip captions and combine these embedding with audio embedding to feed the BiGRU-b…

A Transformer-based Audio Captioning Model with Keyword Estimation

One of the problems with automated audio captioning (AAC) is the indeterminacy in word selection corresponding to the audio event/scene. Since one acoustic event/scene can be described with several words, it results in a combinatorial explosion of possible captions and difficulty in training. To solve this problem, we propose a Transformer-based audio-captioning model with keyword estimation called TRACKE. It simultaneously solves the word-selection indeterminacy problem with the main task of AAC while executing the sub-task of acoustic event detection/acoustic scene classification (i.e., keyw…

Automated Audio Captioning: An Overview of Recent Progress and New Challenges

Automated audio captioning is a cross-modal translation task that aims to generate natural language descriptions for given audio clips. This task has received increasing attention with the release of freely available datasets in recent years. The problem has been addressed predominantly with deep learning techniques. Numerous approaches have been proposed, such as investigating different neural network architectures, exploiting auxiliary information such as keywords or sentence information to guide caption generation, and employing different training strategies, which have greatly facilitated …

零样本音频描述生成：基于音频-语言模型引导与音频上下文关键词

Zero-shot audio captioning with audio-language model guidance and audio context keywords

提出ZerAuCap框架，利用预训练大语言模型和音频-语言模型引导，结合音频上下文关键词，实现零样本音频描述生成，在AudioCaps和Clotho数据集上取得最优结果。

利用辅助AudioSet语义增强音频描述生成

Enhancing Audio Captioning with Auxiliary AudioSet Semantics

提出一种资源高效的自动音频描述生成框架，通过ConvNeXt编码器提取帧级声学表示并融合AudioSet关键词语义，使用紧凑的六层BART解码器生成描述，在Clotho V2和AudioCaps上取得竞争性性能。