多模态动作质量评估
Multimodal Action Quality Assessment
AI总结：
针对动作质量评估仅依赖视觉信息的问题，提出渐进式自适应多模态融合网络（PAMFN），通过模态特定分支与混合模态分支结合及三个新模块，有效融合视觉与音频信息以提升评分准确性。
摘要：
动作质量评估（AQA）是评估动作执行优劣的过程。以往的研究仅利用视觉信息进行建模，忽略了音频信息。我们认为，尽管AQA高度依赖视觉信息，但音频是提高分数回归精度的有用互补信息，特别是对于有背景音乐的体育项目，如花样滑冰和艺术体操。为了利用多模态信息进行AQA，即RGB、光流和音频信息，我们提出了一种渐进式自适应多模态融合网络（PAMFN），该网络分别对模态特定信息和混合模态信息进行建模。我们的模型包含三个独立探索模态特定信息的模态特定分支，以及一个从模态特定分支中渐进式聚合模态特定信息的混合模态分支。为了在模态特定分支和混合模态分支之间建立桥梁，提出了三个新模块。首先，设计了模态特定特征解码器模块，用于选择性地将模态特定信息传递给混合模态分支。其次，在探索模态特定信息之间的交互时，我们认为使用不变的多模态融合策略可能会导致次优结果，从而考虑到动作不同部分的潜在多样性。因此，提出了自适应融合模块来学习动作不同部分的自适应多模态融合策略。该模块由几个用于探索不同多模态融合策略的FusionNet和一个用于决定启用哪些FusionNet的PolicyNet组成。第三，设计了一个名为跨模态特征解码器的模块，用于将自适应融合模块生成的跨模态特征传递给混合模态分支。
英文摘要：
Action quality assessment (AQA) is to assess how well an action is performed. Previous works perform modelling by only the use of visual information, ignoring audio information. We argue that although AQA is highly dependent on visual information, the audio is useful complementary information for improving the score regression accuracy, especially for sports with background music, such as figure skating and rhythmic gymnastics. To leverage multimodal information for AQA, i.e., RGB, optical flow and audio information, we propose a Progressive Adaptive Multimodal Fusion Network (PAMFN) that separately models modality-specific information and mixed-modality information. Our model consists of with three modality-specific branches that independently explore modality-specific information and a mixed-modality branch that progressively aggregates the modality-specific information from the modality-specific branches. To build the bridge between modality-specific branches and the mixed-modality branch, three novel modules are proposed. First, a Modality-specific Feature Decoder module is designed to selectively transfer modality-specific information to the mixed-modality branch. Second, when exploring the interaction between modality-specific information, we argue that using an invariant multimodal fusion policy may lead to suboptimal results, so as to take the potential diversity in different parts of an action into consideration. Therefore, an Adaptive Fusion Module is proposed to learn adaptive multimodal fusion policies in different parts of an action. This module consists of several FusionNets for exploring different multimodal fusion strategies and a PolicyNet for deciding which FusionNets are enabled. Third, a module called Cross-modal Feature Decoder is designed to transfer cross-modal features generated by Adaptive Fusion Module to the mixed-modality branch.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
