arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.01284cs.MMcs.CVcs.SDeess.ASeess.IV
收藏

面向音视频广义零样本学习的分布外检测:一个通用框架

Out-Of-Distribution Detection for Audio-visual Generalized Zero-Shot Learning: A General Framework

School of Physical Sciences University of Science and Technology of China(中国科学技术大学物理科学学院)

Liuyuan Wen

AI总结:

针对音视频广义零样本学习中的生成式训练不稳定和嵌入式领域偏移问题,提出一种结合生成对抗网络与分布外检测的统一框架,通过合成未见特征训练检测器与分类器,在三个数据集上显著优于现有方法。

摘要:

广义零样本学习(GZSL)是一项具有挑战性的任务,要求对已见和未见类别进行准确分类。在该领域内,音视频GZSL由于包含视觉和声学特征作为多模态输入,成为一项极其令人兴奋却又困难的任务。该领域现有的工作大多采用基于嵌入或基于生成的方法。然而,生成式训练困难且不稳定,而基于嵌入的方法常遇到领域偏移问题。因此,我们发现将这两种方法整合到一个统一框架中很有前景,以利用它们的优势并减轻各自的缺点。我们的研究引入了一个采用分布外(OOD)检测的通用框架,旨在发挥这两种方法的优势。我们首先采用生成对抗网络合成未见特征,从而能够训练一个OOD检测器以及已见和未见类别的分类器。该检测器确定测试特征属于已见还是未见类别,随后利用针对每种特征类型的独立分类器进行分类。我们在三个流行的音视频数据集上测试了我们的框架,观察到与现有最先进工作相比有显著改进。代码可在 https://github.com/liuyuan-wen/AV-OOD-GZSL 找到。

英文摘要:

Generalized Zero-Shot Learning (GZSL) is a challenging task requiring accurate classification of both seen and unseen classes. Within this domain, Audio-visual GZSL emerges as an extremely exciting yet difficult task, given the inclusion of both visual and acoustic features as multi-modal inputs. Existing efforts in this field mostly utilize either embedding-based or generative-based methods. However, generative training is difficult and unstable, while embedding-based methods often encounter domain shift problem. Thus, we find it promising to integrate both methods into a unified framework to leverage their advantages while mitigating their respective disadvantages. Our study introduces a general framework employing out-of-distribution (OOD) detection, aiming to harness the strengths of both approaches. We first employ generative adversarial networks to synthesize unseen features, enabling the training of an OOD detector alongside classifiers for seen and unseen classes. This detector determines whether a test feature belongs to seen or unseen classes, followed by classification utilizing separate classifiers for each feature type. We test our framework on three popular audio-visual datasets and observe a significant improvement comparing to existing state-of-the-art works. Codes can be found in https://github.com/liuyuan-wen/AV-OOD-GZSL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑