arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.09833cs.AIcs.MMcs.SDeess.AS
收藏

SHMamba:用于音视频问答的结构化双曲状态空间模型

SHMamba: Structured Hyperbolic State Space Model for Audio-Visual Question Answering

University of Electronic Science and Technology of China(电子科技大学);Harbin Institute of Technology(哈尔滨工业大学)

Zhe Yang, Wenrui Li, Guanghui Cheng

AI总结:

SHMamba提出结构化双曲状态空间模型,融合双曲几何与状态空间模型,解决AVQA中层次结构表示和长序列建模问题,参数减少78.12%,性能提升2.53%。

摘要:

音视频问答(AVQA)任务在应用方面具有巨大潜力。与传统单模态方法相比,AVQA的多模态输入使得特征提取和融合过程更具挑战性。欧几里得空间难以有效表示数据的多维关系,尤其是在提取和处理具有树状结构或层次结构的数据时,欧几里得空间不适合作为嵌入空间。此外,Transformer中的自注意力机制能够有效捕捉序列中元素之间的动态关系,但自注意力机制在窗口建模方面的局限性和二次计算复杂度降低了其在长序列建模中的有效性。为了解决这些局限性,我们提出了SHMamba:结构化双曲状态空间模型,以整合双曲几何和状态空间模型的优势。具体而言,SHMamba利用双曲空间的内在属性来表示音视频数据中的层次结构和复杂关系,同时,状态空间模型通过对整个序列进行全局建模来捕捉随时间变化的动态特征。此外,我们引入了自适应曲率双曲对齐模块和跨模态融合模块,分别增强对层次结构的理解以及跨模态信息的动态交换。大量实验表明,SHMamba在参数和计算成本更少的情况下优于先前的方法。我们的可学习参数减少了78.12%,平均性能提升了2.53%。实验表明,我们的方法在所有当前主流方法中表现出优越性,并且更适合实际应用场景。

英文摘要:

The Audio-Visual Question Answering (AVQA) task holds significant potential for applications. Compared to traditional unimodal approaches, the multi-modal input of AVQA makes feature extraction and fusion processes more challenging. Euclidean space is difficult to effectively represent multi-dimensional relationships of data. Especially when extracting and processing data with a tree structure or hierarchical structure, Euclidean space is not suitable as an embedding space. Additionally, the self-attention mechanism in Transformers is effective in capturing the dynamic relationships between elements in a sequence. However, the self-attention mechanism's limitations in window modeling and quadratic computational complexity reduce its effectiveness in modeling long sequences. To address these limitations, we propose SHMamba: Structured Hyperbolic State Space Model to integrate the advantages of hyperbolic geometry and state space models. Specifically, SHMamba leverages the intrinsic properties of hyperbolic space to represent hierarchical structures and complex relationships in audio-visual data. Meanwhile, the state space model captures dynamic changes over time by globally modeling the entire sequence. Furthermore, we introduce an adaptive curvature hyperbolic alignment module and a cross fusion block to enhance the understanding of hierarchical structures and the dynamic exchange of cross-modal information, respectively. Extensive experiments demonstrate that SHMamba outperforms previous methods with fewer parameters and computational costs. Our learnable parameters are reduced by 78.12\%, while the average performance improves by 2.53\%. Experiments show that our method demonstrates superiority among all current major methods and is more suitable for practical application scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑