arXiv 2606.116022026-06-11cs.CV

面向音视频广义零样本学习的层次化标准化嵌入对齐

On Aligning Hierarchical Standardized Embedding for Audio-visual Generalized Zero-shot Learning

Southeast University（东南大学）；The University of Hong Kong（香港大学）；Beijing Institute of Technology（北京理工大学）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education（新一代人工智能技术及其跨学科应用重点实验室（东南大学），教育部）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）

Zihan Zhang, Jie Hong, Siyuan Fan, Yanghao Zhou, Pengfei Fang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出AHSE方法，通过Z-score标准化和层次化对齐策略（语义、类别、批次三级）解决音视频与文本模态间的分布与结构差异，在三个基准数据集上取得竞争性能。

摘要：

音视频广义零样本学习（AV-GZSL）是一项具有挑战性的任务，旨在通过整合音频和视觉模态的数据来分类已见和未见对象或场景。近期研究主要集中于融合或对齐音频和视觉特征以生成更具信息量的音视频嵌入。此外，大多数现有方法对齐音视频与文本特征仅依赖于优化目标。然而，这些方法忽视了音视频与文本模态之间固有的分布和结构差异。为解决这一局限性，我们提出一种名为层次化标准化嵌入对齐（AHSE）的方法，该方法能够在共享嵌入空间内实现标准化音视频与文本嵌入的层次化对齐。具体而言，我们首先对融合后的音视频和文本嵌入应用Z-score标准化以减少分布不匹配。然后，我们引入一种层次化对齐策略，在语义、类别和批次三个层面最小化差异，从而构建一个更鲁棒且结构良好的嵌入空间。该策略不仅保留了语义和类间关系，还保持了每个批次内的空间一致性。在三个基准数据集：VGGSound-GZSL、UCF-GZSL和ActivityNet-GZSL上的大量实验表明，AHSE在零样本学习中取得了竞争性能。

英文摘要：

Audio-visual Generalized Zero-shot Learning (AV-GZSL) is a challenging task that aims to classify both seen and unseen objects or scenes by integrating data from audio and visual modalities. Recent studies primarily focus on fusing or aligning audio and visual features to generate more informative audio-visual embeddings. Also, aligning the audio-visual and textual features of most existing methods relies solely on the optimization objectives. However, those methods neglect the inherent distributional and structural differences between audio-visual and textual modalities. To address this limitation, we propose a method termed Aligning Hierarchical Standardized Embedding (AHSE), which enables hierarchical alignment of standardized audio-visual and textual embeddings within a shared embedding space. Specifically, we first apply Z-score standardization to the fused audio-visual and textual embeddings to reduce distributional mismatches. We then introduce a hierarchical alignment strategy that minimizes discrepancies at the semantic, class, and batch levels, thereby constructing a more robust and well-structured embedding space. This strategy not only preserves semantic and inter-class relationships but also maintains spatial consistency within each batch. Extensive experiments on three benchmark datasets: VGGSound-GZSL, UCF-GZSL, and ActivityNet-GZSL, demonstrate that AHSE achieves competitive performance in zero-shot learning.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。